2024-03-25T17:00:36.723 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T17:00:36.723 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_amathuri@teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284 -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/orig.config.yaml 2024-03-25T17:00:36.766 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T17:00:36.849 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621284' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQ2cEhYIhasdCSbjjrbMgx4YxiqTKlERSjnvWz3CHmGkjcWVq30z9d3cBowRGiDQ5TgJ7u9R8Ifp+cYmJCQMH0= smithi136.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPAOGTiEUOU1pEwviVaRkC2dui5+QB0HeHVyRSXf0gqxq6Z2loYjA6tAp6VkzPXtjLroU92ffcKbBWZ8ZttYcw= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T17:00:36.849 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T17:00:36.850 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T17:00:36.850 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T17:00:36.852 INFO:teuthology.task.internal:Checking packages... 2024-03-25T17:00:36.874 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T17:00:36.875 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T17:00:36.875 INFO:teuthology.packaging:ref: None 2024-03-25T17:00:36.875 INFO:teuthology.packaging:tag: None 2024-03-25T17:00:36.875 INFO:teuthology.packaging:branch: main 2024-03-25T17:00:36.875 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:00:36.875 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T17:00:37.208 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T17:00:37.209 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T17:00:37.216 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T17:00:37.216 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T17:00:37.250 INFO:teuthology.task.internal:Saving configuration 2024-03-25T17:00:37.266 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T17:00:37.275 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T17:00:37.299 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:55:50.025273', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQ2cEhYIhasdCSbjjrbMgx4YxiqTKlERSjnvWz3CHmGkjcWVq30z9d3cBowRGiDQ5TgJ7u9R8Ifp+cYmJCQMH0='} 2024-03-25T17:00:37.324 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:55:50.023666', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPAOGTiEUOU1pEwviVaRkC2dui5+QB0HeHVyRSXf0gqxq6Z2loYjA6tAp6VkzPXtjLroU92ffcKbBWZ8ZttYcw='} 2024-03-25T17:00:37.324 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T17:00:37.330 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T17:00:37.331 INFO:teuthology.task.internal:roles: ubuntu@smithi136.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T17:00:37.331 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T17:00:37.420 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f734b1200d0>, signals=[15]) 2024-03-25T17:00:37.421 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T17:00:37.427 INFO:teuthology.task.internal:Opening connections... 2024-03-25T17:00:37.427 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T17:00:37.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:00:37.533 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T17:00:37.534 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:00:37.621 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T17:00:37.627 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-03-25T17:00:37.644 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-03-25T17:00:37.644 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-03-25T17:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-03-25T17:00:37.700 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:00:37.701 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:00:37.702 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-03-25T17:00:37.733 DEBUG:teuthology.orchestra.run.smithi136:> uname -m 2024-03-25T17:00:37.750 INFO:teuthology.orchestra.run.smithi136.stdout:x86_64 2024-03-25T17:00:37.750 DEBUG:teuthology.orchestra.run.smithi136:> cat /etc/os-release 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:NAME="CentOS Stream" 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION="9" 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:ID="centos" 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:ID_LIKE="rhel fedora" 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:VERSION_ID="9" 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:PLATFORM_ID="platform:el9" 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:ANSI_COLOR="0;31" 2024-03-25T17:00:37.810 INFO:teuthology.orchestra.run.smithi136.stdout:LOGO="fedora-logo-icon" 2024-03-25T17:00:37.811 INFO:teuthology.orchestra.run.smithi136.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T17:00:37.811 INFO:teuthology.orchestra.run.smithi136.stdout:HOME_URL="https://centos.org/" 2024-03-25T17:00:37.811 INFO:teuthology.orchestra.run.smithi136.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T17:00:37.811 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T17:00:37.811 INFO:teuthology.orchestra.run.smithi136.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T17:00:37.811 INFO:teuthology.lock.ops:Updating smithi136.front.sepia.ceph.com on lock server 2024-03-25T17:00:37.840 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T17:00:37.849 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T17:00:37.875 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T17:00:37.875 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:00:37.878 DEBUG:teuthology.orchestra.run.smithi136:> test '!' -e /home/ubuntu/cephtest 2024-03-25T17:00:37.893 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T17:00:37.899 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T17:00:37.899 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:00:37.935 DEBUG:teuthology.orchestra.run.smithi136:> test -z $(ls -A /var/lib/ceph) 2024-03-25T17:00:38.041 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T17:00:38.095 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T17:00:38.110 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T17:00:38.110 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T17:00:38.110 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:00:38.111 DEBUG:teuthology.orchestra.run.smithi136:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T17:00:38.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:00:38.127 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-25T17:00:38.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:00:38.128 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-25T17:00:38.184 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:00:38.184 INFO:teuthology.task.kernel:Running kernel on smithi136: 5.14.0-361.el9.x86_64 2024-03-25T17:00:38.184 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum install -y kernel 2024-03-25T17:00:38.185 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-361.el9.x86_64 2024-03-25T17:00:38.185 INFO:teuthology.task.kernel:Running kernel on smithi032: 5.14.0-361.el9.x86_64 2024-03-25T17:00:38.186 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum install -y kernel 2024-03-25T17:00:40.384 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T17:00:40.460 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T17:00:41.943 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-25T17:00:42.119 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-25T17:00:43.376 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 8.7 MB/s | 8.0 MB 00:00 2024-03-25T17:00:45.686 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 15 MB/s | 19 MB 00:01 2024-03-25T17:00:46.667 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-25T17:00:49.952 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 8.6 MB/s | 6.1 MB 00:00 2024-03-25T17:00:51.395 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 51 kB/s | 16 kB 00:00 2024-03-25T17:00:52.509 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 4.0 MB/s | 19 MB 00:04 2024-03-25T17:00:57.910 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.1 MB 00:01 2024-03-25T17:00:59.652 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 29 kB/s | 16 kB 00:00 2024-03-25T17:01:01.031 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T17:01:06.171 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-03-25T17:01:08.403 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:01:08.403 INFO:teuthology.orchestra.run.smithi032.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:01:08.503 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:01:08.511 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:01:08.511 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-03-25T17:01:08.511 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:01:08.511 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:01:08.511 INFO:teuthology.orchestra.run.smithi032.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T17:01:08.511 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:01:08.511 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T17:01:08.512 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T17:01:08.512 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T17:01:08.512 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:01:08.512 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:01:08.512 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:01:08.512 INFO:teuthology.orchestra.run.smithi032.stdout:Install 4 Packages 2024-03-25T17:01:08.512 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:01:08.512 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 76 M 2024-03-25T17:01:08.513 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 123 M 2024-03-25T17:01:08.513 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:01:08.906 INFO:teuthology.orchestra.run.smithi032.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 400 kB/s | 44 kB 00:00 2024-03-25T17:01:09.656 INFO:teuthology.orchestra.run.smithi032.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 17 MB/s | 15 MB 00:00 2024-03-25T17:01:12.298 INFO:teuthology.orchestra.run.smithi032.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 8.1 MB/s | 27 MB 00:03 2024-03-25T17:01:12.598 INFO:teuthology.orchestra.run.smithi032.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 8.8 MB/s | 33 MB 00:03 2024-03-25T17:01:12.598 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:01:12.598 INFO:teuthology.orchestra.run.smithi032.stdout:Total 19 MB/s | 76 MB 00:04 2024-03-25T17:01:12.645 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:21 2024-03-25T17:01:13.293 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:01:13.722 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:01:13.722 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:01:14.918 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:01:14.918 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:01:17.528 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:01:17.634 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-03-25T17:01:18.196 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T17:01:18.283 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T17:01:19.799 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T17:01:19.799 INFO:teuthology.orchestra.run.smithi136.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T17:01:19.903 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:01:19.908 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:01:19.909 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:01:19.909 INFO:teuthology.orchestra.run.smithi136.stdout:Install 4 Packages 2024-03-25T17:01:19.909 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:01:19.909 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 76 M 2024-03-25T17:01:19.909 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 123 M 2024-03-25T17:01:19.909 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:01:20.302 INFO:teuthology.orchestra.run.smithi136.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 146 kB/s | 44 kB 00:00 2024-03-25T17:01:21.137 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T17:01:21.269 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T17:01:21.969 INFO:teuthology.orchestra.run.smithi136.stdout:(2/4): kernel-modules-core-5.14.0-430.el9.x86_6 16 MB/s | 27 MB 00:01 2024-03-25T17:01:22.119 INFO:teuthology.orchestra.run.smithi136.stdout:(3/4): kernel-core-5.14.0-430.el9.x86_64.rpm 7.0 MB/s | 15 MB 00:02 2024-03-25T17:01:22.553 INFO:teuthology.orchestra.run.smithi136.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 13 MB/s | 33 MB 00:02 2024-03-25T17:01:22.553 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:01:22.553 INFO:teuthology.orchestra.run.smithi136.stdout:Total 29 MB/s | 76 MB 00:02 2024-03-25T17:01:23.215 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:01:23.603 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:01:23.604 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:01:24.590 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:01:24.590 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:01:26.329 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T17:01:26.363 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:01:27.057 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:01:27.667 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T17:01:27.755 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T17:01:30.480 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T17:01:30.609 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T17:01:31.458 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:01:35.316 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T17:01:35.417 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:01:40.135 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:02.104 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:02.105 INFO:teuthology.orchestra.run.smithi032.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:02:02.105 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:02:02.149 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:02.467 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:02.467 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T17:02:02.468 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T17:02:02.468 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T17:02:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:02.901 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:02:02.902 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:02:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T17:02:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T17:02:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T17:02:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T17:02:02.902 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:02:02.902 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:02:03.215 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:02:03.310 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T17:02:03.311 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T17:02:03.311 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T17:02:03.311 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:02:03.311 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:02:03.311 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T17:02:03.311 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:02:03.311 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:02:03.393 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T17:02:03.393 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T17:02:03.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 68.el9 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8264254 2024-03-25T17:02:03.430 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:02:03.431 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:02:03.432 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:02:03.432 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 5.14.0-430.el9.x86_64 2024-03-25T17:02:03.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:02:04.108 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-03-25T17:02:05.627 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:02:05.646 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-03-25T17:02:05.648 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-25T17:02:05.663 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.W9SkTPdrvY 2024-03-25T17:02:05.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.W9SkTPdrvY 2024-03-25T17:02:05.742 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.W9SkTPdrvY 2024-03-25T17:02:05.882 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.W9SkTPdrvY is 6KB 2024-03-25T17:02:05.931 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.W9SkTPdrvY 2024-03-25T17:02:05.946 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:02:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:02:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:02:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:02:06.015 INFO:teuthology.orchestra.run.smithi032.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T17:02:06.016 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T17:02:06.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-03-25T17:02:09.679 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:09.679 INFO:teuthology.orchestra.run.smithi136.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T17:02:09.679 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:02:09.723 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:10.158 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:10.159 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T17:02:10.159 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T17:02:10.159 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:02:10.558 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:02:10.901 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:02:10.997 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T17:02:10.997 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T17:02:10.998 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T17:02:10.998 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T17:02:10.998 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T17:02:10.998 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T17:02:10.998 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi136.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T17:02:10.998 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T17:02:11.078 INFO:teuthology.orchestra.run.smithi136.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T17:02:11.078 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T17:02:11.078 DEBUG:teuthology.orchestra.run.smithi136:> sudo rpm -qi grub2-tools 2024-03-25T17:02:11.112 INFO:teuthology.orchestra.run.smithi136.stdout:Name : grub2-tools 2024-03-25T17:02:11.112 INFO:teuthology.orchestra.run.smithi136.stdout:Epoch : 1 2024-03-25T17:02:11.112 INFO:teuthology.orchestra.run.smithi136.stdout:Version : 2.06 2024-03-25T17:02:11.112 INFO:teuthology.orchestra.run.smithi136.stdout:Release : 68.el9 2024-03-25T17:02:11.112 INFO:teuthology.orchestra.run.smithi136.stdout:Architecture: x86_64 2024-03-25T17:02:11.112 INFO:teuthology.orchestra.run.smithi136.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T17:02:11.112 INFO:teuthology.orchestra.run.smithi136.stdout:Group : Unspecified 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Size : 8264254 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:License : GPLv3+ 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Packager : builder@centos.org 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Vendor : CentOS 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Summary : Support tools for GRUB. 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:Description : 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T17:02:11.113 INFO:teuthology.orchestra.run.smithi136.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T17:02:11.114 INFO:teuthology.orchestra.run.smithi136.stdout:hardware devices. 2024-03-25T17:02:11.114 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:02:11.114 INFO:teuthology.orchestra.run.smithi136.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T17:02:11.114 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T17:02:11.114 INFO:teuthology.task.kernel:Updating grub on smithi136 to boot 5.14.0-430.el9.x86_64 2024-03-25T17:02:11.115 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T17:02:11.717 INFO:teuthology.orchestra.run.smithi136.stderr:Generating grub configuration file ... 2024-03-25T17:02:13.193 INFO:teuthology.orchestra.run.smithi136.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T17:02:13.225 INFO:teuthology.orchestra.run.smithi136.stderr:done 2024-03-25T17:02:13.228 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-25T17:02:13.243 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.jkrdV0WgEF 2024-03-25T17:02:13.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.jkrdV0WgEF 2024-03-25T17:02:13.312 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.jkrdV0WgEF 2024-03-25T17:02:13.451 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.jkrdV0WgEF is 6KB 2024-03-25T17:02:13.500 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.jkrdV0WgEF 2024-03-25T17:02:13.514 DEBUG:teuthology.orchestra.run.smithi136:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T17:02:13.578 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T17:02:13.579 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T17:02:13.579 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T17:02:13.579 INFO:teuthology.orchestra.run.smithi136.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T17:02:13.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T17:02:13.731 DEBUG:teuthology.orchestra.run.smithi136:> sudo shutdown -r now 2024-03-25T17:02:36.182 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T17:02:36.183 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-25T17:02:36.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:02:43.739 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T17:02:43.739 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-25T17:02:43.740 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:02:54.750 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-03-25T17:03:02.174 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-25T17:03:03.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-25T17:03:03.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:03:11.182 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-25T17:03:11.183 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:03:14.238 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.136 2024-03-25T17:03:22.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-03-25T17:03:26.242 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-25T17:03:26.243 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:03:34.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-25T17:03:34.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:03:34.509 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-25T17:03:34.895 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-25T17:03:34.895 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T17:03:34.895 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-03-25T17:03:34.909 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-430.el9.x86_64 2024-03-25T17:03:34.909 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T17:03:34.909 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:03:34.909 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:03:35.910 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T17:03:35.910 DEBUG:teuthology.orchestra.run.smithi032:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:03:35.982 INFO:teuthology.orchestra.run.smithi032.stdout:ttyS1 2024-03-25T17:03:36.033 DEBUG:teuthology.parallel:result is None 2024-03-25T17:03:58.091 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-25T17:03:58.767 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-25T17:03:58.767 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T17:03:58.768 DEBUG:teuthology.orchestra.run.smithi136:> uname -r 2024-03-25T17:03:58.796 INFO:teuthology.orchestra.run.smithi136.stdout:5.14.0-430.el9.x86_64 2024-03-25T17:03:58.796 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T17:03:58.797 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T17:03:58.797 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T17:03:59.797 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T17:03:59.798 DEBUG:teuthology.orchestra.run.smithi136:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T17:03:59.958 INFO:teuthology.orchestra.run.smithi136.stdout:ttyS1 2024-03-25T17:03:59.996 DEBUG:teuthology.parallel:result is None 2024-03-25T17:03:59.996 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T17:04:00.003 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T17:04:00.004 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:04:00.007 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T17:04:00.023 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T17:04:00.050 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T17:04:00.083 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T17:04:00.083 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:04:00.085 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T17:04:00.120 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T17:04:00.150 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T17:04:00.150 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:04:00.152 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T17:04:00.188 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:04:00.198 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:04:00.201 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:04:00.211 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T17:04:00.213 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T17:04:00.219 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T17:04:00.220 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:04:00.245 DEBUG:teuthology.orchestra.run.smithi136:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T17:04:00.283 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T17:04:00.292 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T17:04:00.292 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:04:00.315 DEBUG:teuthology.orchestra.run.smithi136:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T17:04:00.342 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:04:00.414 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:04:00.464 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:04:00.547 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:04:00.589 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T17:04:00.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:04:00.656 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:04:00.696 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T17:04:00.748 DEBUG:teuthology.orchestra.run.smithi136:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:04:21.598 DEBUG:teuthology.orchestra.run.smithi136:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T17:04:21.631 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T17:04:21.632 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T17:04:21.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-03-25T17:04:21.705 DEBUG:teuthology.orchestra.run.smithi136:> sudo service rsyslog restart 2024-03-25T17:04:21.753 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:04:21.867 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T17:04:22.073 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T17:04:22.081 INFO:teuthology.task.internal:Starting timer... 2024-03-25T17:04:22.081 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T17:04:22.129 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T17:04:22.160 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-03-25T17:04:22.221 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-03-25T17:04:22.224 DEBUG:teuthology.orchestra.run.smithi136:> sudo service auditd rotate 2024-03-25T17:04:22.301 INFO:teuthology.orchestra.run.smithi136.stdout:Rotating logs: 2024-03-25T17:04:22.303 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T17:04:22.304 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-03-25T17:04:22.327 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-03-25T17:04:22.328 DEBUG:teuthology.orchestra.run.smithi136:> /usr/sbin/getenforce 2024-03-25T17:04:22.351 INFO:teuthology.orchestra.run.smithi136.stdout:Permissive 2024-03-25T17:04:22.351 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi136.front.sepia.ceph.com': 'permissive'} 2024-03-25T17:04:22.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:04:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:04:22.397 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T17:04:22.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T17:04:22.426 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T17:04:22.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:04:22.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo /usr/sbin/setenforce permissive 2024-03-25T17:04:22.511 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T17:04:22.521 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-25T17:04:22.528 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-25T17:04:22.679 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T17:04:22.694 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T17:04:22.695 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi136.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T17:08:36.628 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi136.front.sepia.ceph.com')] 2024-03-25T17:08:36.629 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-25T17:08:36.631 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:08:36.720 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-03-25T17:08:36.797 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-03-25T17:08:36.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-25T17:08:36.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi136.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T17:08:36.883 DEBUG:teuthology.orchestra.run.smithi136:> true 2024-03-25T17:08:36.961 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi136.front.sepia.ceph.com' 2024-03-25T17:08:36.961 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T17:08:36.972 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T17:08:36.972 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:08:36.973 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:08:36.978 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T17:08:36.978 DEBUG:teuthology.orchestra.run.smithi136:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T17:08:37.004 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:08:37.021 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:08:37.031 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T17:08:37.047 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T17:08:37.050 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-03-25T17:08:37.064 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-03-25T17:08:37.076 INFO:teuthology.orchestra.run.smithi136.stderr:sudo: ntpd: command not found 2024-03-25T17:08:37.081 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:08:37.090 INFO:teuthology.orchestra.run.smithi136.stdout:506 Cannot talk to daemon 2024-03-25T17:08:37.097 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:08:37.107 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T17:08:37.123 INFO:teuthology.orchestra.run.smithi136.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T17:08:37.152 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:08:37.155 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:08:37.155 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-03-25T17:08:37.155 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:08:37.155 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:08:37.155 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:08:37.155 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:08:37.176 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-25T17:08:37.181 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T17:08:37.181 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-25T17:08:37.181 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:08:37.181 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:08:37.181 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:08:37.181 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T17:08:37.182 INFO:teuthology.run_tasks:Running task install... 2024-03-25T17:08:37.190 DEBUG:teuthology.task.install:project ceph 2024-03-25T17:08:37.190 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T17:08:37.190 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T17:08:37.190 INFO:teuthology.task.install:Using flavor: default 2024-03-25T17:08:37.196 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T17:08:37.196 INFO:teuthology.task.install:extra packages: [] 2024-03-25T17:08:37.196 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:08:37.197 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:08:37.199 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T17:08:37.199 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T17:08:37.420 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:08:37.420 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:08:37.423 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T17:08:37.423 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T17:08:37.766 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:08:37.766 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T17:08:37.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:08:37.796 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:08:37.796 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:08:37.824 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T17:08:37.825 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T17:08:37.825 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T17:08:37.854 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T17:08:37.854 DEBUG:teuthology.orchestra.run.smithi136:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T17:08:37.865 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:08:37.923 DEBUG:teuthology.orchestra.run.smithi136:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T17:08:37.946 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:08:38.004 DEBUG:teuthology.orchestra.run.smithi136:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T17:08:38.013 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-03-25T17:08:38.014 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-03-25T17:08:38.071 INFO:teuthology.orchestra.run.smithi136.stdout:check_obsoletes = 1 2024-03-25T17:08:38.072 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum clean all 2024-03-25T17:08:38.256 INFO:teuthology.orchestra.run.smithi032.stdout:74 files removed 2024-03-25T17:08:38.283 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw 2024-03-25T17:08:38.313 INFO:teuthology.orchestra.run.smithi136.stdout:74 files removed 2024-03-25T17:08:38.340 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw 2024-03-25T17:08:39.580 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 83 kB/s | 82 kB 00:00 2024-03-25T17:08:39.595 INFO:teuthology.orchestra.run.smithi136.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-25T17:08:40.413 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T17:08:40.479 INFO:teuthology.orchestra.run.smithi136.stdout:ceph noarch packages 12 kB/s | 10 kB 00:00 2024-03-25T17:08:41.271 INFO:teuthology.orchestra.run.smithi136.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-03-25T17:08:41.331 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 2.2 kB/s | 1.9 kB 00:00 2024-03-25T17:08:41.758 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - AppStream 17 MB/s | 6.7 MB 00:00 2024-03-25T17:08:41.898 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T17:08:43.273 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS-9 - BaseOS 7.4 MB/s | 2.0 MB 00:00 2024-03-25T17:08:43.341 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS-9 - BaseOS 4.8 MB/s | 2.0 MB 00:00 2024-03-25T17:08:44.239 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-25T17:08:44.524 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 9.6 MB/s | 8.0 MB 00:00 2024-03-25T17:08:46.990 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-03-25T17:08:48.802 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 5.9 MB/s | 19 MB 00:03 2024-03-25T17:08:51.257 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - CRB 7.6 MB/s | 6.1 MB 00:00 2024-03-25T17:08:52.599 INFO:teuthology.orchestra.run.smithi136.stdout:CentOS Stream 9 - Extras packages 87 kB/s | 16 kB 00:00 2024-03-25T17:08:54.753 INFO:teuthology.orchestra.run.smithi136.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-25T17:08:59.642 INFO:teuthology.orchestra.run.smithi136.stdout:lab-extras 45 kB/s | 1.7 kB 00:00 2024-03-25T17:09:00.358 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 798 kB/s | 6.1 MB 00:07 2024-03-25T17:09:01.876 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-25T17:09:02.085 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:09:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:09:02.089 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:09:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:09:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:09:02.089 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:09:02.089 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrading: 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:09:02.090 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:09:02.091 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:09:02.092 INFO:teuthology.orchestra.run.smithi136.stdout:Installing weak dependencies: 2024-03-25T17:09:02.093 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T17:09:02.093 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:09:02.093 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:09:02.093 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:09:02.093 INFO:teuthology.orchestra.run.smithi136.stdout:Install 30 Packages 2024-03-25T17:09:02.093 INFO:teuthology.orchestra.run.smithi136.stdout:Upgrade 2 Packages 2024-03-25T17:09:02.093 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:09:02.094 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 58 M 2024-03-25T17:09:02.094 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:09:06.263 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 4.8 MB/s | 21 MB 00:04 2024-03-25T17:09:11.328 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-25T17:09:13.942 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:09:13.966 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:09:13.966 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:09:13.966 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:09:13.966 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:09:13.966 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T17:09:13.967 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T17:09:13.968 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout:Install 30 Packages 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 2 Packages 2024-03-25T17:09:13.969 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:09:13.970 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 58 M 2024-03-25T17:09:13.970 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:09:27.530 INFO:teuthology.orchestra.run.smithi136.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 219 kB/s | 5.3 MB 00:24 2024-03-25T17:09:27.780 INFO:teuthology.orchestra.run.smithi136.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 101 kB/s | 25 kB 00:00 2024-03-25T17:09:32.429 INFO:teuthology.orchestra.run.smithi136.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 160 kB/s | 742 kB 00:04 2024-03-25T17:09:34.536 INFO:teuthology.orchestra.run.smithi032.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 269 kB/s | 5.3 MB 00:20 2024-03-25T17:09:34.579 INFO:teuthology.orchestra.run.smithi136.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 232 kB/s | 498 kB 00:02 2024-03-25T17:09:34.652 INFO:teuthology.orchestra.run.smithi032.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-25T17:09:38.494 INFO:teuthology.orchestra.run.smithi032.stdout:(3/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 410 kB/s | 9.7 MB 00:24 2024-03-25T17:09:38.836 INFO:teuthology.orchestra.run.smithi032.stdout:(4/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 177 kB/s | 742 kB 00:04 2024-03-25T17:09:39.503 INFO:teuthology.orchestra.run.smithi032.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 494 kB/s | 498 kB 00:01 2024-03-25T17:09:39.886 INFO:teuthology.orchestra.run.smithi032.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 118 kB/s | 45 kB 00:00 2024-03-25T17:09:40.387 INFO:teuthology.orchestra.run.smithi032.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 263 kB/s | 131 kB 00:00 2024-03-25T17:09:41.045 INFO:teuthology.orchestra.run.smithi032.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 246 kB/s | 162 kB 00:00 2024-03-25T17:09:42.162 INFO:teuthology.orchestra.run.smithi136.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 9.7 MB 00:39 2024-03-25T17:09:42.254 INFO:teuthology.orchestra.run.smithi032.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 265 kB/s | 320 kB 00:01 2024-03-25T17:09:42.562 INFO:teuthology.orchestra.run.smithi136.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 113 kB/s | 45 kB 00:00 2024-03-25T17:09:43.154 INFO:teuthology.orchestra.run.smithi032.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 331 kB/s | 298 kB 00:00 2024-03-25T17:09:43.287 INFO:teuthology.orchestra.run.smithi136.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 181 kB/s | 131 kB 00:00 2024-03-25T17:09:44.063 INFO:teuthology.orchestra.run.smithi032.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 109 kB/s | 99 kB 00:00 2024-03-25T17:09:44.171 INFO:teuthology.orchestra.run.smithi032.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 985 kB/s | 106 kB 00:00 2024-03-25T17:09:44.213 INFO:teuthology.orchestra.run.smithi032.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T17:09:44.247 INFO:teuthology.orchestra.run.smithi032.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T17:09:44.305 INFO:teuthology.orchestra.run.smithi032.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T17:09:44.347 INFO:teuthology.orchestra.run.smithi032.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-25T17:09:44.389 INFO:teuthology.orchestra.run.smithi032.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 989 kB/s | 41 kB 00:00 2024-03-25T17:09:44.620 INFO:teuthology.orchestra.run.smithi136.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 121 kB/s | 162 kB 00:01 2024-03-25T17:09:44.623 INFO:teuthology.orchestra.run.smithi032.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 929 kB/s | 216 kB 00:00 2024-03-25T17:09:44.681 INFO:teuthology.orchestra.run.smithi032.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 598 kB/s | 35 kB 00:00 2024-03-25T17:09:44.740 INFO:teuthology.orchestra.run.smithi032.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 764 kB/s | 44 kB 00:00 2024-03-25T17:09:45.563 INFO:teuthology.orchestra.run.smithi032.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 375 kB/s | 309 kB 00:00 2024-03-25T17:09:46.282 INFO:teuthology.orchestra.run.smithi032.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 6.1 MB/s | 4.4 MB 00:00 2024-03-25T17:09:46.441 INFO:teuthology.orchestra.run.smithi032.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 164 kB/s | 26 kB 00:00 2024-03-25T17:09:46.599 INFO:teuthology.orchestra.run.smithi032.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 307 kB/s | 49 kB 00:00 2024-03-25T17:09:46.664 INFO:teuthology.orchestra.run.smithi136.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 157 kB/s | 320 kB 00:02 2024-03-25T17:09:46.749 INFO:teuthology.orchestra.run.smithi032.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 449 kB/s | 67 kB 00:00 2024-03-25T17:09:46.925 INFO:teuthology.orchestra.run.smithi032.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 838 kB 00:00 2024-03-25T17:09:47.091 INFO:teuthology.orchestra.run.smithi032.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-25T17:09:48.428 INFO:teuthology.orchestra.run.smithi136.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 169 kB/s | 298 kB 00:01 2024-03-25T17:09:48.929 INFO:teuthology.orchestra.run.smithi136.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 199 kB/s | 99 kB 00:00 2024-03-25T17:09:49.037 INFO:teuthology.orchestra.run.smithi136.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T17:09:49.070 INFO:teuthology.orchestra.run.smithi136.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 4.7 MB/s | 159 kB 00:00 2024-03-25T17:09:49.096 INFO:teuthology.orchestra.run.smithi136.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.7 MB/s | 44 kB 00:00 2024-03-25T17:09:49.155 INFO:teuthology.orchestra.run.smithi136.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T17:09:49.196 INFO:teuthology.orchestra.run.smithi136.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T17:09:49.230 INFO:teuthology.orchestra.run.smithi136.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T17:09:49.497 INFO:teuthology.orchestra.run.smithi136.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 813 kB/s | 216 kB 00:00 2024-03-25T17:09:49.580 INFO:teuthology.orchestra.run.smithi136.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 418 kB/s | 35 kB 00:00 2024-03-25T17:09:49.639 INFO:teuthology.orchestra.run.smithi136.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 764 kB/s | 44 kB 00:00 2024-03-25T17:09:50.017 INFO:teuthology.orchestra.run.smithi032.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 554 kB/s | 1.6 MB 00:02 2024-03-25T17:09:50.322 INFO:teuthology.orchestra.run.smithi136.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 452 kB/s | 309 kB 00:00 2024-03-25T17:09:50.639 INFO:teuthology.orchestra.run.smithi136.stdout:(22/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 325 kB/s | 5.1 MB 00:16 2024-03-25T17:09:50.931 INFO:teuthology.orchestra.run.smithi136.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 89 kB/s | 26 kB 00:00 2024-03-25T17:09:51.140 INFO:teuthology.orchestra.run.smithi136.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-03-25T17:09:51.349 INFO:teuthology.orchestra.run.smithi136.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2024-03-25T17:09:51.640 INFO:teuthology.orchestra.run.smithi136.stdout:(26/32): libarrow-9.0.0-10.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-03-25T17:09:51.766 INFO:teuthology.orchestra.run.smithi136.stdout:(27/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.0 MB/s | 838 kB 00:00 2024-03-25T17:09:51.791 INFO:teuthology.orchestra.run.smithi136.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-03-25T17:09:52.366 INFO:teuthology.orchestra.run.smithi136.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 2.6 MB/s | 1.6 MB 00:00 2024-03-25T17:09:58.991 INFO:teuthology.orchestra.run.smithi032.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 385 kB/s | 3.4 MB 00:08 2024-03-25T17:09:59.616 INFO:teuthology.orchestra.run.smithi032.stdout:(30/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 251 kB/s | 5.1 MB 00:20 2024-03-25T17:10:06.381 INFO:teuthology.orchestra.run.smithi136.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 237 kB/s | 3.4 MB 00:14 2024-03-25T17:10:06.441 INFO:teuthology.orchestra.run.smithi032.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 430 kB/s | 3.1 MB 00:07 2024-03-25T17:10:12.188 INFO:teuthology.orchestra.run.smithi136.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 292 kB/s | 20 MB 01:09 2024-03-25T17:10:15.637 INFO:teuthology.orchestra.run.smithi136.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 138 kB/s | 3.1 MB 00:23 2024-03-25T17:10:15.640 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:10:15.641 INFO:teuthology.orchestra.run.smithi136.stdout:Total 805 kB/s | 58 MB 01:13 2024-03-25T17:10:15.676 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:10:15.715 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:10:15.715 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:10:16.190 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:10:16.190 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:10:17.656 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:10:17.748 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T17:10:17.783 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T17:10:17.818 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T17:10:17.856 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T17:10:17.874 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T17:10:18.040 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T17:10:18.095 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T17:10:18.420 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T17:10:18.469 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:10:18.544 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:10:18.634 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T17:10:18.662 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:10:18.724 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:10:18.753 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T17:10:18.797 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T17:10:18.865 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:10:18.889 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:10:18.951 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:10:18.979 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T17:10:19.089 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T17:10:19.122 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T17:10:19.162 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T17:10:19.194 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T17:10:19.223 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T17:10:19.256 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T17:10:19.574 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T17:10:19.603 INFO:teuthology.orchestra.run.smithi136.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:10:19.692 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:10:19.724 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T17:10:19.765 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T17:10:20.224 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T17:10:20.246 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:10:20.309 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:10:20.378 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T17:10:20.407 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T17:10:22.489 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:10:22.520 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:10:23.074 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:10:23.114 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:10:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:10:23.198 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T17:10:23.198 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T17:10:23.198 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:10:23.224 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:10:28.573 INFO:teuthology.orchestra.run.smithi032.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 274 kB/s | 20 MB 01:14 2024-03-25T17:10:28.576 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:10:28.577 INFO:teuthology.orchestra.run.smithi032.stdout:Total 794 kB/s | 58 MB 01:14 2024-03-25T17:10:28.607 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:10:28.648 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:10:28.648 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:10:29.142 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:10:29.142 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:10:30.744 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:10:30.839 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T17:10:30.869 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T17:10:30.894 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T17:10:30.925 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T17:10:30.941 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T17:10:31.142 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T17:10:31.198 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T17:10:31.541 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T17:10:31.588 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:10:31.688 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:10:31.777 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T17:10:31.806 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:10:31.852 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T17:10:31.881 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T17:10:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T17:10:31.994 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:10:32.017 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:10:32.062 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T17:10:32.081 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T17:10:32.192 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T17:10:32.242 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T17:10:32.282 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T17:10:32.314 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T17:10:32.343 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T17:10:32.376 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T17:10:32.605 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:10:32.606 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /sys 2024-03-25T17:10:32.606 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /proc 2024-03-25T17:10:32.606 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /mnt 2024-03-25T17:10:32.606 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /var/tmp 2024-03-25T17:10:32.606 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /home 2024-03-25T17:10:32.606 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /root 2024-03-25T17:10:32.606 INFO:teuthology.orchestra.run.smithi136.stdout:skipping the directory /tmp 2024-03-25T17:10:32.606 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:10:32.706 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T17:10:32.731 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:10:32.804 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T17:10:32.835 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T17:10:32.876 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T17:10:33.359 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T17:10:33.383 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:10:33.438 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T17:10:33.500 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T17:10:33.521 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:10:33.535 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T17:10:33.546 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:10:33.546 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:10:33.546 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T17:10:33.546 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:10:33.546 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:10:33.546 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:10:33.548 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:10:33.624 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:10:33.624 INFO:teuthology.orchestra.run.smithi136.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T17:10:34.743 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T17:10:34.744 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T17:10:34.745 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T17:10:34.745 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T17:10:34.745 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T17:10:34.745 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:10:34.745 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T17:10:34.745 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T17:10:35.821 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:10:35.857 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:10:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:10:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:10:35.885 INFO:teuthology.orchestra.run.smithi136.stdout:Upgraded: 2024-03-25T17:10:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.885 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T17:10:35.886 INFO:teuthology.orchestra.run.smithi136.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T17:10:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T17:10:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T17:10:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T17:10:35.888 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:10:35.888 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:10:36.247 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test 2024-03-25T17:10:36.436 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T17:10:36.477 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:10:36.578 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T17:10:36.578 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T17:10:36.578 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T17:10:36.578 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:10:36.603 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:10:36.807 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:01:37 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:10:36.957 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:10:36.958 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:10:36.958 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:10:36.958 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:10:36.958 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:10:36.958 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T17:10:36.958 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-03-25T17:10:36.959 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:10:36.960 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 48 M 2024-03-25T17:10:36.960 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 204 M 2024-03-25T17:10:36.960 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:10:37.264 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T17:10:37.314 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-25T17:10:37.347 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T17:10:38.214 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 216 kB/s | 187 kB 00:00 2024-03-25T17:10:38.231 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 144 kB/s | 163 kB 00:01 2024-03-25T17:10:46.427 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:10:46.428 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-03-25T17:10:46.428 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-03-25T17:10:46.428 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-03-25T17:10:46.428 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-03-25T17:10:46.428 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-03-25T17:10:46.428 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-03-25T17:10:46.428 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-03-25T17:10:46.428 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:10:47.357 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:10:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T17:10:47.386 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:10:47.386 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T17:10:47.386 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:10:47.386 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T17:10:47.386 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:10:47.388 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:10:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T17:10:47.476 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:10:48.594 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:10:48.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T17:10:48.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T17:10:48.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T17:10:48.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T17:10:48.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T17:10:48.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T17:10:48.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T17:10:48.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T17:10:48.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T17:10:48.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T17:10:49.560 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T17:10:49.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:10:49.560 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T17:10:49.561 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.562 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:49.563 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T17:10:49.563 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T17:10:49.563 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T17:10:49.563 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T17:10:49.563 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T17:10:49.563 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:10:49.563 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:10:49.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test 2024-03-25T17:10:50.422 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:01:39 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:10:50.576 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:10:50.577 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:10:50.577 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:10:50.577 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:10:50.577 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-03-25T17:10:50.578 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:10:50.579 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 48 M 2024-03-25T17:10:50.579 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 204 M 2024-03-25T17:10:50.579 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:10:50.898 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 855 kB/s | 217 kB 00:00 2024-03-25T17:10:51.040 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 2.1 MB/s | 300 kB 00:00 2024-03-25T17:10:51.090 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-25T17:10:51.416 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 576 kB/s | 187 kB 00:00 2024-03-25T17:10:52.049 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 116 kB/s | 163 kB 00:01 2024-03-25T17:12:30.144 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 428 kB/s | 47 MB 01:53 2024-03-25T17:12:30.146 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:12:30.146 INFO:teuthology.orchestra.run.smithi136.stdout:Total 436 kB/s | 48 MB 01:53 2024-03-25T17:12:30.157 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:12:30.177 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:12:30.177 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:12:30.556 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:12:30.556 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:12:31.144 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:12:31.226 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T17:12:31.263 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T17:12:31.295 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:12:31.344 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T17:12:31.371 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:12:35.810 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:12:35.887 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:12:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:12:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T17:12:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T17:12:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:12:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T17:12:38.307 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T17:12:38.729 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T17:12:38.729 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:12:38.729 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:12:38.730 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:12:38.730 INFO:teuthology.orchestra.run.smithi136.stdout: jq-1.6-16.el9.x86_64 2024-03-25T17:12:38.730 INFO:teuthology.orchestra.run.smithi136.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:12:38.730 INFO:teuthology.orchestra.run.smithi136.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T17:12:38.730 INFO:teuthology.orchestra.run.smithi136.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T17:12:38.730 INFO:teuthology.orchestra.run.smithi136.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T17:12:38.730 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:12:38.730 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:12:39.064 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph 2024-03-25T17:12:39.634 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:03:40 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:12:39.793 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T17:12:39.796 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T17:12:39.797 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T17:12:39.798 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout:========================================================================================= 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout:Install 36 Packages 2024-03-25T17:12:39.799 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:12:39.800 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 30 M 2024-03-25T17:12:39.800 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 107 M 2024-03-25T17:12:39.800 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:12:40.839 INFO:teuthology.orchestra.run.smithi136.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T17:12:45.513 INFO:teuthology.orchestra.run.smithi136.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 296 kB/s | 1.5 MB 00:05 2024-03-25T17:12:49.079 INFO:teuthology.orchestra.run.smithi136.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 249 kB/s | 2.1 MB 00:08 2024-03-25T17:12:49.779 INFO:teuthology.orchestra.run.smithi136.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 354 kB/s | 247 kB 00:00 2024-03-25T17:12:49.929 INFO:teuthology.orchestra.run.smithi136.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-25T17:12:49.971 INFO:teuthology.orchestra.run.smithi136.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T17:12:50.005 INFO:teuthology.orchestra.run.smithi136.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 946 kB/s | 32 kB 00:00 2024-03-25T17:12:50.038 INFO:teuthology.orchestra.run.smithi136.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T17:12:50.072 INFO:teuthology.orchestra.run.smithi136.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T17:12:50.114 INFO:teuthology.orchestra.run.smithi136.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T17:12:50.181 INFO:teuthology.orchestra.run.smithi136.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T17:12:50.215 INFO:teuthology.orchestra.run.smithi136.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T17:12:50.248 INFO:teuthology.orchestra.run.smithi136.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T17:12:50.274 INFO:teuthology.orchestra.run.smithi136.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 7.3 MB/s | 187 kB 00:00 2024-03-25T17:12:50.399 INFO:teuthology.orchestra.run.smithi136.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-25T17:12:50.558 INFO:teuthology.orchestra.run.smithi136.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 141 kB/s | 22 kB 00:00 2024-03-25T17:12:51.099 INFO:teuthology.orchestra.run.smithi136.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 279 kB/s | 151 kB 00:00 2024-03-25T17:12:51.216 INFO:teuthology.orchestra.run.smithi136.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 372 kB/s | 43 kB 00:00 2024-03-25T17:12:51.383 INFO:teuthology.orchestra.run.smithi136.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-03-25T17:12:51.584 INFO:teuthology.orchestra.run.smithi136.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T17:12:51.675 INFO:teuthology.orchestra.run.smithi136.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 117 kB/s | 11 kB 00:00 2024-03-25T17:12:51.767 INFO:teuthology.orchestra.run.smithi136.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T17:12:51.868 INFO:teuthology.orchestra.run.smithi136.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 231 kB/s | 23 kB 00:00 2024-03-25T17:12:51.968 INFO:teuthology.orchestra.run.smithi136.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 195 kB/s | 19 kB 00:00 2024-03-25T17:12:52.060 INFO:teuthology.orchestra.run.smithi136.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 215 kB/s | 20 kB 00:00 2024-03-25T17:12:52.152 INFO:teuthology.orchestra.run.smithi136.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 504 kB/s | 46 kB 00:00 2024-03-25T17:12:52.261 INFO:teuthology.orchestra.run.smithi136.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 728 kB/s | 79 kB 00:00 2024-03-25T17:12:52.395 INFO:teuthology.orchestra.run.smithi136.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.0 MB/s | 272 kB 00:00 2024-03-25T17:12:52.504 INFO:teuthology.orchestra.run.smithi136.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 150 kB/s | 16 kB 00:00 2024-03-25T17:12:52.612 INFO:teuthology.orchestra.run.smithi136.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 841 kB/s | 90 kB 00:00 2024-03-25T17:12:52.712 INFO:teuthology.orchestra.run.smithi136.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 357 kB/s | 36 kB 00:00 2024-03-25T17:12:52.837 INFO:teuthology.orchestra.run.smithi136.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-25T17:12:52.979 INFO:teuthology.orchestra.run.smithi136.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.9 MB/s | 427 kB 00:00 2024-03-25T17:12:53.080 INFO:teuthology.orchestra.run.smithi136.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 199 kB/s | 20 kB 00:00 2024-03-25T17:12:54.613 INFO:teuthology.orchestra.run.smithi136.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 348 kB/s | 4.7 MB 00:13 2024-03-25T17:13:07.402 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 354 kB/s | 47 MB 02:16 2024-03-25T17:13:07.404 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:07.405 INFO:teuthology.orchestra.run.smithi032.stdout:Total 361 kB/s | 48 MB 02:16 2024-03-25T17:13:07.416 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:13:07.437 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:13:07.437 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:13:07.826 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:13:07.827 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:13:08.436 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:13:08.636 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T17:13:08.677 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T17:13:08.717 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:13:08.769 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T17:13:08.796 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:13:13.522 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T17:13:13.598 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:13:15.938 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T17:13:15.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T17:13:15.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T17:13:15.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T17:13:15.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T17:13:15.938 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T17:13:16.316 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T17:13:16.316 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:13:16.316 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:13:16.316 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:16.317 INFO:teuthology.orchestra.run.smithi032.stdout: jq-1.6-16.el9.x86_64 2024-03-25T17:13:16.317 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:16.317 INFO:teuthology.orchestra.run.smithi032.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T17:13:16.317 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T17:13:16.317 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T17:13:16.317 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:13:16.317 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:13:16.622 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph 2024-03-25T17:13:17.194 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:04:06 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:13:17.361 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:13:17.363 INFO:teuthology.orchestra.run.smithi032.stdout:========================================================================================= 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout:========================================================================================= 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T17:13:17.364 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T17:13:17.365 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:13:17.366 INFO:teuthology.orchestra.run.smithi032.stdout:========================================================================================= 2024-03-25T17:13:17.367 INFO:teuthology.orchestra.run.smithi032.stdout:Install 36 Packages 2024-03-25T17:13:17.367 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:13:17.367 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 30 M 2024-03-25T17:13:17.367 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 107 M 2024-03-25T17:13:17.367 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:13:18.148 INFO:teuthology.orchestra.run.smithi032.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 17 kB/s | 6.6 kB 00:00 2024-03-25T17:13:24.856 INFO:teuthology.orchestra.run.smithi032.stdout:(2/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 715 kB/s | 4.7 MB 00:06 2024-03-25T17:13:25.456 INFO:teuthology.orchestra.run.smithi032.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 198 kB/s | 1.5 MB 00:07 2024-03-25T17:13:26.082 INFO:teuthology.orchestra.run.smithi032.stdout:(4/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 261 kB/s | 2.1 MB 00:08 2024-03-25T17:13:26.199 INFO:teuthology.orchestra.run.smithi032.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T17:13:26.232 INFO:teuthology.orchestra.run.smithi032.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 4.2 MB/s | 144 kB 00:00 2024-03-25T17:13:26.266 INFO:teuthology.orchestra.run.smithi032.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 952 kB/s | 32 kB 00:00 2024-03-25T17:13:26.299 INFO:teuthology.orchestra.run.smithi032.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T17:13:26.333 INFO:teuthology.orchestra.run.smithi032.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T17:13:26.375 INFO:teuthology.orchestra.run.smithi032.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T17:13:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T17:13:26.484 INFO:teuthology.orchestra.run.smithi032.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T17:13:26.517 INFO:teuthology.orchestra.run.smithi032.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.7 MB/s | 124 kB 00:00 2024-03-25T17:13:26.551 INFO:teuthology.orchestra.run.smithi032.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T17:13:26.618 INFO:teuthology.orchestra.run.smithi032.stdout:(15/36): ceph-mgr-modules-core-19.0.0-1439.g5dc 213 kB/s | 247 kB 00:01 2024-03-25T17:13:26.668 INFO:teuthology.orchestra.run.smithi032.stdout:(16/36): python3-requests-2.25.1-8.el9.noarch.r 1.0 MB/s | 125 kB 00:00 2024-03-25T17:13:27.094 INFO:teuthology.orchestra.run.smithi032.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 355 kB/s | 151 kB 00:00 2024-03-25T17:13:27.186 INFO:teuthology.orchestra.run.smithi032.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 474 kB/s | 43 kB 00:00 2024-03-25T17:13:27.211 INFO:teuthology.orchestra.run.smithi032.stdout:(19/36): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-03-25T17:13:27.328 INFO:teuthology.orchestra.run.smithi032.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-03-25T17:13:27.411 INFO:teuthology.orchestra.run.smithi032.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 128 kB/s | 11 kB 00:00 2024-03-25T17:13:27.503 INFO:teuthology.orchestra.run.smithi032.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T17:13:27.587 INFO:teuthology.orchestra.run.smithi032.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 278 kB/s | 23 kB 00:00 2024-03-25T17:13:27.662 INFO:teuthology.orchestra.run.smithi032.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 259 kB/s | 19 kB 00:00 2024-03-25T17:13:27.696 INFO:teuthology.orchestra.run.smithi032.stdout:(25/36): python3-cherrypy-18.6.1-2.el9.noarch.r 739 kB/s | 358 kB 00:00 2024-03-25T17:13:27.738 INFO:teuthology.orchestra.run.smithi032.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 262 kB/s | 20 kB 00:00 2024-03-25T17:13:27.780 INFO:teuthology.orchestra.run.smithi032.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 554 kB/s | 46 kB 00:00 2024-03-25T17:13:27.846 INFO:teuthology.orchestra.run.smithi032.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 726 kB/s | 79 kB 00:00 2024-03-25T17:13:27.897 INFO:teuthology.orchestra.run.smithi032.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-25T17:13:27.922 INFO:teuthology.orchestra.run.smithi032.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 219 kB/s | 16 kB 00:00 2024-03-25T17:13:27.989 INFO:teuthology.orchestra.run.smithi032.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 980 kB/s | 90 kB 00:00 2024-03-25T17:13:28.006 INFO:teuthology.orchestra.run.smithi032.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 427 kB/s | 36 kB 00:00 2024-03-25T17:13:28.098 INFO:teuthology.orchestra.run.smithi032.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-25T17:13:28.181 INFO:teuthology.orchestra.run.smithi032.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.4 MB/s | 427 kB 00:00 2024-03-25T17:13:28.198 INFO:teuthology.orchestra.run.smithi032.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-25T17:13:34.630 INFO:teuthology.orchestra.run.smithi136.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 350 kB/s | 17 MB 00:49 2024-03-25T17:13:34.635 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:34.635 INFO:teuthology.orchestra.run.smithi136.stdout:Total 554 kB/s | 30 MB 00:54 2024-03-25T17:13:34.657 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:13:34.699 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:13:34.699 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:13:35.130 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:13:35.130 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:13:36.472 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:13:36.541 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T17:13:36.582 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T17:13:36.610 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T17:13:36.679 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T17:13:36.717 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T17:13:36.750 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T17:13:36.841 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T17:13:36.919 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T17:13:36.961 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T17:13:37.032 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T17:13:37.086 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T17:13:37.125 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T17:13:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T17:13:37.229 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T17:13:37.420 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T17:13:37.463 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T17:13:37.494 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T17:13:37.545 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T17:13:37.575 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T17:13:37.650 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T17:13:37.685 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:13:37.719 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T17:13:37.759 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T17:13:37.794 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T17:13:37.829 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T17:13:37.917 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T17:13:37.951 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T17:13:38.023 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T17:13:38.159 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T17:13:38.276 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:13:38.435 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T17:13:38.457 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:13:38.483 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:13:38.483 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:13:38.483 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T17:13:38.483 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:13:38.483 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:13:38.483 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:39.983 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:13:40.011 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:13:40.011 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:13:40.012 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T17:13:40.012 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:13:40.012 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:13:40.012 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:40.451 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:13:40.474 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:13:40.475 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:13:40.475 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T17:13:40.475 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:13:40.475 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:13:40.475 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:40.685 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:13:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:13:40.708 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:13:40.708 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T17:13:40.708 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:13:40.708 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:13:40.708 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:40.774 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:13:42.268 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T17:13:42.269 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T17:13:42.270 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T17:13:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T17:13:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T17:13:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:13:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T17:13:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T17:13:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T17:13:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T17:13:42.271 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T17:13:43.224 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T17:13:43.224 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:43.224 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:13:43.224 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:43.224 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:43.224 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:43.224 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T17:13:43.225 INFO:teuthology.orchestra.run.smithi136.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T17:13:43.226 INFO:teuthology.orchestra.run.smithi136.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T17:13:43.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T17:13:43.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T17:13:43.227 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:43.227 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:13:43.578 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base 2024-03-25T17:13:44.149 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:13:44.246 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:13:44.298 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:13:44.300 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:13:44.300 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:13:44.370 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install cephadm 2024-03-25T17:13:44.932 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:13:45.082 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:13:45.082 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 747 k 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 754 k 2024-03-25T17:13:45.083 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:13:47.640 INFO:teuthology.orchestra.run.smithi136.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 292 kB/s | 747 kB 00:02 2024-03-25T17:13:47.641 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:47.641 INFO:teuthology.orchestra.run.smithi136.stdout:Total 292 kB/s | 747 kB 00:02 2024-03-25T17:13:47.641 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:13:47.647 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:13:47.647 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:13:47.653 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:13:47.654 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:13:47.728 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:13:48.229 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:13:48.320 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:13:48.728 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:13:48.990 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:13:48.991 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:48.991 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:13:48.991 INFO:teuthology.orchestra.run.smithi136.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:13:48.991 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:48.991 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:13:49.221 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T17:13:49.807 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:50 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:13:49.961 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:13:49.961 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:13:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:13:49.961 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:13:49.961 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:13:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T17:13:49.961 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:49.962 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:13:49.962 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:13:49.962 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:13:49.962 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:49.962 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 145 k 2024-03-25T17:13:49.962 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 436 k 2024-03-25T17:13:49.962 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:13:51.114 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 126 kB/s | 145 kB 00:01 2024-03-25T17:13:51.114 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:13:51.114 INFO:teuthology.orchestra.run.smithi136.stdout:Total 126 kB/s | 145 kB 00:01 2024-03-25T17:13:51.115 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:13:51.124 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:13:51.124 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:13:51.180 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:13:51.180 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:13:51.325 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:13:51.373 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:13:51.398 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:13:51.398 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:13:51.398 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T17:13:51.398 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:52.373 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:13:52.373 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:52.373 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:13:52.373 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:13:52.373 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:52.373 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:13:52.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr 2024-03-25T17:13:53.247 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:54 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:13:53.348 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:13:53.399 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:13:53.400 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:13:53.400 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:13:53.483 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T17:13:54.043 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:04:55 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:13:54.193 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:13:54.194 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-25T17:13:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:13:54.194 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-25T17:13:54.194 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:13:54.194 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T17:13:54.194 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout:====================================================================================== 2024-03-25T17:13:54.195 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-03-25T17:13:54.196 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:13:54.196 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 6.9 M 2024-03-25T17:13:54.196 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-03-25T17:13:54.196 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:13:54.946 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 53 kB/s | 31 kB 00:00 2024-03-25T17:13:54.976 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 25 kB/s | 15 kB 00:00 2024-03-25T17:13:55.110 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-25T17:13:55.169 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.5 MB/s | 1.0 MB 00:00 2024-03-25T17:13:55.327 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.9 MB/s | 863 kB 00:00 2024-03-25T17:13:55.569 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 80 kB/s | 19 kB 00:00 2024-03-25T17:13:55.845 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 811 kB/s | 548 kB 00:00 2024-03-25T17:13:55.953 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-25T17:13:56.054 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 308 kB/s | 31 kB 00:00 2024-03-25T17:13:56.170 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-25T17:13:56.221 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.1 MB/s | 2.0 MB 00:00 2024-03-25T17:14:00.837 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 281 kB/s | 1.8 MB 00:06 2024-03-25T17:14:00.841 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:00.841 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.0 MB/s | 6.9 MB 00:06 2024-03-25T17:14:00.860 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:14:00.879 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:14:00.879 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:14:01.063 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:14:01.063 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:14:01.574 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:14:01.654 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T17:14:01.697 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T17:14:01.743 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T17:14:01.774 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T17:14:01.876 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T17:14:02.008 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T17:14:02.110 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T17:14:02.201 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T17:14:02.258 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:14:02.290 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T17:14:02.531 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T17:14:02.557 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T17:14:03.331 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T17:14:03.332 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:14:03.332 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T17:14:03.332 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T17:14:03.688 INFO:teuthology.orchestra.run.smithi136.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T17:14:03.689 INFO:teuthology.orchestra.run.smithi136.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T17:14:03.689 INFO:teuthology.orchestra.run.smithi136.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T17:14:03.689 INFO:teuthology.orchestra.run.smithi136.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T17:14:03.689 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:03.689 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:14:03.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T17:14:04.505 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:05:05 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:14:04.656 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:14:04.657 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-25T17:14:04.657 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:14:04.657 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-25T17:14:04.657 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:14:04.657 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T17:14:04.657 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:14:04.657 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:14:04.658 INFO:teuthology.orchestra.run.smithi136.stdout:============================================================================================ 2024-03-25T17:14:04.659 INFO:teuthology.orchestra.run.smithi136.stdout:Install 12 Packages 2024-03-25T17:14:04.659 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:04.659 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 37 M 2024-03-25T17:14:04.659 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 209 M 2024-03-25T17:14:04.659 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:14:04.952 INFO:teuthology.orchestra.run.smithi136.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 272 kB/s | 32 kB 00:00 2024-03-25T17:14:05.002 INFO:teuthology.orchestra.run.smithi136.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 639 kB/s | 17 kB 00:00 2024-03-25T17:14:05.028 INFO:teuthology.orchestra.run.smithi136.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T17:14:05.178 INFO:teuthology.orchestra.run.smithi136.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 32 MB/s | 4.7 MB 00:00 2024-03-25T17:14:05.303 INFO:teuthology.orchestra.run.smithi136.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 39 MB/s | 4.8 MB 00:00 2024-03-25T17:14:05.345 INFO:teuthology.orchestra.run.smithi136.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T17:14:05.412 INFO:teuthology.orchestra.run.smithi136.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.2 MB/s | 3.0 MB 00:00 2024-03-25T17:14:05.696 INFO:teuthology.orchestra.run.smithi136.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-03-25T17:14:05.896 INFO:teuthology.orchestra.run.smithi136.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-25T17:14:05.921 INFO:teuthology.orchestra.run.smithi136.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 868 kB/s | 195 kB 00:00 2024-03-25T17:14:06.138 INFO:teuthology.orchestra.run.smithi136.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.0 MB/s | 245 kB 00:00 2024-03-25T17:14:14.202 INFO:teuthology.orchestra.run.smithi032.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 348 kB/s | 17 MB 00:49 2024-03-25T17:14:14.207 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:14.207 INFO:teuthology.orchestra.run.smithi032.stdout:Total 534 kB/s | 30 MB 00:56 2024-03-25T17:14:14.228 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:14:14.272 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:14:14.272 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:14:14.679 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:14:14.680 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:14:16.003 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:14:16.071 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T17:14:16.113 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T17:14:16.140 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T17:14:16.215 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T17:14:16.256 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T17:14:16.289 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T17:14:16.376 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T17:14:16.453 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T17:14:16.500 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T17:14:16.574 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T17:14:16.626 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T17:14:16.664 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T17:14:16.729 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T17:14:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T17:14:16.981 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T17:14:17.026 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T17:14:17.064 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T17:14:17.099 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T17:14:17.129 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T17:14:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T17:14:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:14:17.264 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T17:14:17.305 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T17:14:17.340 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T17:14:17.374 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T17:14:17.494 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T17:14:17.538 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T17:14:17.614 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T17:14:17.755 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T17:14:17.879 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:14:18.045 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T17:14:18.067 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:14:18.092 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T17:14:18.092 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:14:18.092 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T17:14:18.092 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:14:18.092 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T17:14:18.093 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:19.644 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:14:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T17:14:19.672 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:14:19.672 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T17:14:19.672 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:14:19.672 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T17:14:19.672 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:20.145 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:14:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T17:14:20.168 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:14:20.168 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T17:14:20.168 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:14:20.168 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T17:14:20.168 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:14:20.409 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T17:14:20.409 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:14:20.410 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T17:14:20.410 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:14:20.410 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T17:14:20.410 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:20.479 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:14:22.019 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T17:14:22.019 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T17:14:22.019 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T17:14:22.019 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T17:14:22.019 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T17:14:22.019 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T17:14:22.019 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T17:14:22.020 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T17:14:22.021 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:23.180 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T17:14:23.181 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T17:14:23.182 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T17:14:23.183 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:23.183 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:14:23.515 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base 2024-03-25T17:14:24.094 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:14:24.194 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:14:24.248 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:14:24.249 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:14:24.250 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:14:24.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install cephadm 2024-03-25T17:14:24.886 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:14:25.040 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:14:25.041 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:14:25.041 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 747 k 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 754 k 2024-03-25T17:14:25.042 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:14:28.242 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 234 kB/s | 747 kB 00:03 2024-03-25T17:14:28.242 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:28.243 INFO:teuthology.orchestra.run.smithi032.stdout:Total 233 kB/s | 747 kB 00:03 2024-03-25T17:14:28.243 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:14:28.250 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:14:28.250 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:14:28.256 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:14:28.256 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:14:28.330 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:14:28.815 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:14:28.909 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:14:29.292 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:14:29.583 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T17:14:29.583 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:29.583 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:14:29.583 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:29.583 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:29.583 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:14:29.797 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T17:14:30.378 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:14:30.534 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:14:30.534 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:14:30.534 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:14:30.534 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:14:30.534 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:14:30.534 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T17:14:30.534 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:30.534 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:14:30.535 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:14:30.535 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:14:30.535 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:30.535 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 145 k 2024-03-25T17:14:30.535 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 436 k 2024-03-25T17:14:30.535 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:14:31.423 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 164 kB/s | 145 kB 00:00 2024-03-25T17:14:31.423 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:31.424 INFO:teuthology.orchestra.run.smithi032.stdout:Total 163 kB/s | 145 kB 00:00 2024-03-25T17:14:31.424 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:14:31.433 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:14:31.434 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:14:31.493 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:14:31.494 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:14:31.639 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:14:31.698 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:14:31.722 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:14:31.722 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:14:31.723 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T17:14:31.723 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:32.690 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T17:14:32.690 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:32.690 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:14:32.690 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:14:32.690 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:32.690 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:14:32.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr 2024-03-25T17:14:33.500 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:05:22 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:14:33.602 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:14:33.656 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:14:33.658 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:14:33.658 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:14:33.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T17:14:34.279 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:05:23 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:14:34.434 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:14:34.435 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================== 2024-03-25T17:14:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:14:34.435 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================== 2024-03-25T17:14:34.435 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:14:34.435 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T17:14:34.435 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T17:14:34.436 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:34.437 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:14:34.437 INFO:teuthology.orchestra.run.smithi032.stdout:====================================================================================== 2024-03-25T17:14:34.437 INFO:teuthology.orchestra.run.smithi032.stdout:Install 12 Packages 2024-03-25T17:14:34.437 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:34.437 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 6.9 M 2024-03-25T17:14:34.437 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 26 M 2024-03-25T17:14:34.438 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:14:35.017 INFO:teuthology.orchestra.run.smithi032.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 32 kB/s | 15 kB 00:00 2024-03-25T17:14:35.167 INFO:teuthology.orchestra.run.smithi032.stdout:(2/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-03-25T17:14:35.192 INFO:teuthology.orchestra.run.smithi032.stdout:(3/12): ceph-grafana-dashboards-19.0.0-1439.g5d 48 kB/s | 31 kB 00:00 2024-03-25T17:14:35.218 INFO:teuthology.orchestra.run.smithi032.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.6 MB/s | 238 kB 00:00 2024-03-25T17:14:35.526 INFO:teuthology.orchestra.run.smithi032.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.7 MB/s | 548 kB 00:00 2024-03-25T17:14:35.568 INFO:teuthology.orchestra.run.smithi032.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 468 kB/s | 19 kB 00:00 2024-03-25T17:14:35.610 INFO:teuthology.orchestra.run.smithi032.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 2.0 MB/s | 863 kB 00:00 2024-03-25T17:14:35.719 INFO:teuthology.orchestra.run.smithi032.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 14 MB/s | 2.0 MB 00:00 2024-03-25T17:14:35.769 INFO:teuthology.orchestra.run.smithi032.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 614 kB/s | 31 kB 00:00 2024-03-25T17:14:35.811 INFO:teuthology.orchestra.run.smithi032.stdout:(10/12): python3-grpcio-tools-1.46.7-10.el9.x86 721 kB/s | 144 kB 00:00 2024-03-25T17:14:35.836 INFO:teuthology.orchestra.run.smithi032.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.8 MB/s | 188 kB 00:00 2024-03-25T17:14:36.791 INFO:teuthology.orchestra.run.smithi136.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 237 kB/s | 7.4 MB 00:31 2024-03-25T17:14:36.795 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:36.795 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.2 MB/s | 37 MB 00:32 2024-03-25T17:14:36.835 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:14:36.860 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:14:36.860 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:14:37.213 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:14:37.214 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:14:37.949 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:14:38.022 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T17:14:38.062 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T17:14:38.154 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:14:38.190 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T17:14:38.367 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:14:38.389 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:14:38.478 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T17:14:39.129 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T17:14:39.219 INFO:teuthology.orchestra.run.smithi032.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 390 kB/s | 1.8 MB 00:04 2024-03-25T17:14:39.223 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:39.223 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.4 MB/s | 6.9 MB 00:04 2024-03-25T17:14:39.244 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:14:39.244 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T17:14:39.264 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:14:39.264 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:14:39.441 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:14:39.441 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:14:39.964 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:14:40.038 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T17:14:40.072 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T17:14:40.119 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T17:14:40.149 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T17:14:40.259 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T17:14:40.387 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T17:14:40.463 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T17:14:40.481 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T17:14:40.529 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T17:14:40.557 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:14:40.589 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T17:14:40.842 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T17:14:40.880 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:14:41.483 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T17:14:41.515 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:14:41.682 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T17:14:41.683 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T17:14:41.684 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T17:14:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T17:14:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:42.032 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:14:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T17:14:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:42.032 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:42.033 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:14:42.256 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T17:14:42.825 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:05:31 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:14:42.982 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T17:14:42.984 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout:============================================================================================ 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout:Install 12 Packages 2024-03-25T17:14:42.985 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:14:42.986 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 37 M 2024-03-25T17:14:42.986 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 209 M 2024-03-25T17:14:42.986 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:14:43.209 INFO:teuthology.orchestra.run.smithi032.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 296 kB/s | 32 kB 00:00 2024-03-25T17:14:43.259 INFO:teuthology.orchestra.run.smithi032.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 669 kB/s | 17 kB 00:00 2024-03-25T17:14:43.284 INFO:teuthology.orchestra.run.smithi032.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T17:14:43.443 INFO:teuthology.orchestra.run.smithi032.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 30 MB/s | 4.7 MB 00:00 2024-03-25T17:14:43.576 INFO:teuthology.orchestra.run.smithi032.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T17:14:43.618 INFO:teuthology.orchestra.run.smithi032.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.3 MB/s | 217 kB 00:00 2024-03-25T17:14:43.660 INFO:teuthology.orchestra.run.smithi032.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.4 MB/s | 3.0 MB 00:00 2024-03-25T17:14:43.986 INFO:teuthology.orchestra.run.smithi032.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-25T17:14:44.018 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T17:14:44.019 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T17:14:44.036 INFO:teuthology.orchestra.run.smithi032.stdout:(9/12): libgfortran-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 803 kB 00:00 2024-03-25T17:14:44.144 INFO:teuthology.orchestra.run.smithi032.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-03-25T17:14:44.295 INFO:teuthology.orchestra.run.smithi032.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 949 kB/s | 245 kB 00:00 2024-03-25T17:14:44.476 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T17:14:44.476 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:44.476 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:14:44.476 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:44.476 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T17:14:44.476 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T17:14:44.476 INFO:teuthology.orchestra.run.smithi136.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T17:14:44.476 INFO:teuthology.orchestra.run.smithi136.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:44.477 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:14:44.887 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-rook 2024-03-25T17:14:45.461 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:05:46 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:14:45.612 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:14:45.613 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:14:45.613 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:14:45.613 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T17:14:45.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout:Install 13 Packages 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 2.0 M 2024-03-25T17:14:45.615 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 26 M 2024-03-25T17:14:45.616 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:14:46.057 INFO:teuthology.orchestra.run.smithi136.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 243 kB/s | 24 kB 00:00 2024-03-25T17:14:46.108 INFO:teuthology.orchestra.run.smithi136.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 120 kB/s | 18 kB 00:00 2024-03-25T17:14:46.149 INFO:teuthology.orchestra.run.smithi136.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T17:14:46.175 INFO:teuthology.orchestra.run.smithi136.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-25T17:14:46.200 INFO:teuthology.orchestra.run.smithi136.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T17:14:46.217 INFO:teuthology.orchestra.run.smithi136.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 1.0 MB/s | 43 kB 00:00 2024-03-25T17:14:46.451 INFO:teuthology.orchestra.run.smithi136.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-03-25T17:14:46.468 INFO:teuthology.orchestra.run.smithi136.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 121 kB/s | 32 kB 00:00 2024-03-25T17:14:46.602 INFO:teuthology.orchestra.run.smithi136.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 81 kB/s | 52 kB 00:00 2024-03-25T17:14:46.827 INFO:teuthology.orchestra.run.smithi136.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 540 kB/s | 203 kB 00:00 2024-03-25T17:14:46.902 INFO:teuthology.orchestra.run.smithi136.stdout:(11/13): python3-kubernetes-26.1.0-2.el9.noarch 2.4 MB/s | 1.0 MB 00:00 2024-03-25T17:14:46.927 INFO:teuthology.orchestra.run.smithi136.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 182 kB/s | 59 kB 00:00 2024-03-25T17:14:46.952 INFO:teuthology.orchestra.run.smithi136.stdout:(13/13): python3-websocket-client-1.2.3-2.el9.n 715 kB/s | 90 kB 00:00 2024-03-25T17:14:46.954 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:46.955 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.5 MB/s | 2.0 MB 00:01 2024-03-25T17:14:46.955 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:14:46.968 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:14:46.968 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:14:47.057 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:14:47.058 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:14:47.563 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:14:47.722 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T17:14:47.770 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T17:14:47.825 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T17:14:47.854 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T17:14:47.895 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T17:14:47.989 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T17:14:48.097 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T17:14:48.157 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T17:14:48.870 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T17:14:48.913 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T17:14:48.947 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T17:14:48.995 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T17:14:49.017 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:14:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:14:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T17:14:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T17:14:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T17:14:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T17:14:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T17:14:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T17:14:49.915 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T17:14:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T17:14:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T17:14:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T17:14:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T17:14:49.916 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T17:14:50.260 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T17:14:50.260 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:50.260 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:14:50.260 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:50.260 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:50.261 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:14:50.518 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T17:14:51.084 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:05:52 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:14:51.234 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:14:51.235 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:14:51.235 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:14:51.235 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:14:51.235 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout:Install 6 Packages 2024-03-25T17:14:51.236 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:51.237 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 6.8 M 2024-03-25T17:14:51.237 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 33 M 2024-03-25T17:14:51.237 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:14:51.849 INFO:teuthology.orchestra.run.smithi136.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T17:14:52.307 INFO:teuthology.orchestra.run.smithi136.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.2 MB/s | 548 kB 00:00 2024-03-25T17:14:52.407 INFO:teuthology.orchestra.run.smithi136.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 577 kB/s | 58 kB 00:00 2024-03-25T17:14:52.508 INFO:teuthology.orchestra.run.smithi136.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 509 kB/s | 51 kB 00:00 2024-03-25T17:14:52.541 INFO:teuthology.orchestra.run.smithi136.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 257 kB/s | 248 kB 00:00 2024-03-25T17:14:52.583 INFO:teuthology.orchestra.run.smithi136.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 137 kB/s | 138 kB 00:01 2024-03-25T17:14:52.585 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:14:52.585 INFO:teuthology.orchestra.run.smithi136.stdout:Total 5.0 MB/s | 6.8 MB 00:01 2024-03-25T17:14:52.597 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:14:52.607 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:14:52.607 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:14:52.696 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:14:52.698 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:14:53.016 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:14:53.164 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T17:14:53.219 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T17:14:53.731 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T17:14:53.813 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T17:14:53.875 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T17:14:53.897 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:14:54.696 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:14:54.696 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T17:14:54.697 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T17:14:54.697 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T17:14:54.697 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T17:14:54.697 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T17:14:54.993 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:54.994 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:14:55.280 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse 2024-03-25T17:14:55.851 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:05:56 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:14:56.003 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:14:56.003 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:14:56.003 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:14:56.003 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:14:56.003 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:14:56.004 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T17:14:56.004 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:14:56.004 INFO:teuthology.orchestra.run.smithi136.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T17:14:56.004 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:56.004 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:14:56.004 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:14:56.004 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-25T17:14:56.004 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:14:56.005 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 964 k 2024-03-25T17:14:56.005 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.8 M 2024-03-25T17:14:56.005 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:14:56.152 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 533 kB/s | 79 kB 00:00 2024-03-25T17:15:00.127 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 215 kB/s | 885 kB 00:04 2024-03-25T17:15:00.128 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:00.129 INFO:teuthology.orchestra.run.smithi136.stdout:Total 234 kB/s | 964 kB 00:04 2024-03-25T17:15:00.129 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:15:00.151 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:15:00.151 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:15:00.227 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:15:00.228 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:15:00.409 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:15:00.561 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T17:15:00.627 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:15:01.356 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:15:01.357 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:15:01.692 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T17:15:01.692 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:01.692 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:15:01.692 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T17:15:01.692 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:01.692 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:02.000 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-volume 2024-03-25T17:15:02.583 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:03 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:02.734 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:02.734 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repository Size 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-25T17:15:02.735 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:02.736 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 332 k 2024-03-25T17:15:02.736 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 1.5 M 2024-03-25T17:15:02.736 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:15:03.015 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 246 kB/s | 69 kB 00:00 2024-03-25T17:15:04.199 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 180 kB/s | 264 kB 00:01 2024-03-25T17:15:04.201 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:04.201 INFO:teuthology.orchestra.run.smithi136.stdout:Total 227 kB/s | 332 kB 00:01 2024-03-25T17:15:04.201 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:15:04.211 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:15:04.211 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:15:04.258 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:15:04.258 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:15:04.408 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:15:04.581 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T17:15:04.610 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:15:04.625 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:15:04.625 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:15:04.626 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T17:15:04.626 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:05.403 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T17:15:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T17:15:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:05.733 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:15:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:15:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T17:15:05.733 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:05.733 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:06.075 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel 2024-03-25T17:15:06.658 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:07 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:06.807 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:15:06.808 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:06.809 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 127 k 2024-03-25T17:15:06.809 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 456 k 2024-03-25T17:15:06.809 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:15:07.559 INFO:teuthology.orchestra.run.smithi032.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 309 kB/s | 7.4 MB 00:24 2024-03-25T17:15:07.562 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:07.562 INFO:teuthology.orchestra.run.smithi032.stdout:Total 1.5 MB/s | 37 MB 00:24 2024-03-25T17:15:07.606 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:15:07.631 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:15:07.631 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:15:07.813 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 126 kB/s | 127 kB 00:01 2024-03-25T17:15:07.814 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:07.814 INFO:teuthology.orchestra.run.smithi136.stdout:Total 126 kB/s | 127 kB 00:01 2024-03-25T17:15:07.814 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:15:07.820 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:15:07.821 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:15:07.879 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:15:07.879 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:15:08.001 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:15:08.002 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:15:08.032 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:15:08.129 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:08.589 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:08.742 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:15:08.816 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T17:15:08.847 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T17:15:08.865 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:08.865 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:08.865 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:15:08.865 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:08.865 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:08.865 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:08.946 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:15:08.984 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T17:15:09.084 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2 2024-03-25T17:15:09.169 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:15:09.199 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:15:09.285 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T17:15:09.663 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:09.762 INFO:teuthology.orchestra.run.smithi136.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:09.819 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:09.820 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:15:09.821 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:09.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs-devel 2024-03-25T17:15:09.976 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T17:15:10.098 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T17:15:10.456 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:11 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:10.604 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:10.605 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:10.605 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:15:10.605 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:10.605 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:15:10.605 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T17:15:10.606 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:10.606 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:15:10.606 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:10.606 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:15:10.606 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:10.606 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 31 k 2024-03-25T17:15:10.606 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 140 k 2024-03-25T17:15:10.606 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:15:11.234 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 50 kB/s | 31 kB 00:00 2024-03-25T17:15:11.235 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:11.235 INFO:teuthology.orchestra.run.smithi136.stdout:Total 49 kB/s | 31 kB 00:00 2024-03-25T17:15:11.235 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:15:11.239 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:15:11.239 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:15:11.267 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:15:11.267 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:15:11.397 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T17:15:11.410 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:15:11.525 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:11.895 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:12.156 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:12.156 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:12.156 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:15:12.157 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:12.157 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:12.157 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:12.334 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2 2024-03-25T17:15:12.490 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T17:15:12.514 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:15:12.912 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:13 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:13.016 INFO:teuthology.orchestra.run.smithi136.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:13.071 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:13.072 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:15:13.072 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:13.133 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1 2024-03-25T17:15:13.736 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:14 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:13.835 INFO:teuthology.orchestra.run.smithi136.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:13.888 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:13.889 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:15:13.889 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:13.943 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados 2024-03-25T17:15:14.508 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:14.604 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:14.656 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:14.657 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:15:14.657 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:14.719 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw 2024-03-25T17:15:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T17:15:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T17:15:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T17:15:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T17:15:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T17:15:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T17:15:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T17:15:15.003 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T17:15:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T17:15:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T17:15:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T17:15:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T17:15:15.283 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:16 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:15.380 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:15.406 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T17:15:15.406 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:15.406 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:15:15.406 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:15:15.406 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T17:15:15.406 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T17:15:15.406 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:15.407 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:15.433 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:15.434 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:15:15.434 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:15.490 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs 2024-03-25T17:15:15.820 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-rook 2024-03-25T17:15:16.059 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:16.155 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:16.207 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:16.208 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:15:16.208 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:16.261 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd 2024-03-25T17:15:16.401 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:05 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:16.557 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:16.558 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:15:16.558 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T17:15:16.559 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout:Install 13 Packages 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 2.0 M 2024-03-25T17:15:16.560 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 26 M 2024-03-25T17:15:16.561 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:15:16.747 INFO:teuthology.orchestra.run.smithi032.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 227 kB/s | 24 kB 00:00 2024-03-25T17:15:16.805 INFO:teuthology.orchestra.run.smithi032.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 109 kB/s | 18 kB 00:00 2024-03-25T17:15:16.823 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:17 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:16.831 INFO:teuthology.orchestra.run.smithi032.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 3.0 MB/s | 174 kB 00:00 2024-03-25T17:15:16.856 INFO:teuthology.orchestra.run.smithi032.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-25T17:15:16.881 INFO:teuthology.orchestra.run.smithi032.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-25T17:15:16.906 INFO:teuthology.orchestra.run.smithi032.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 871 kB/s | 43 kB 00:00 2024-03-25T17:15:16.920 INFO:teuthology.orchestra.run.smithi136.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:16.972 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:16.973 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T17:15:16.973 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:16.982 INFO:teuthology.orchestra.run.smithi032.stdout:(7/13): python3-certifi-2021.10.8-2.el9.noarch. 199 kB/s | 15 kB 00:00 2024-03-25T17:15:17.007 INFO:teuthology.orchestra.run.smithi032.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-03-25T17:15:17.027 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse 2024-03-25T17:15:17.166 INFO:teuthology.orchestra.run.smithi032.stdout:(9/13): python3-google-auth-2.28.2-1.el9.noarch 1.1 MB/s | 203 kB 00:00 2024-03-25T17:15:17.207 INFO:teuthology.orchestra.run.smithi032.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 1.4 MB/s | 59 kB 00:00 2024-03-25T17:15:17.258 INFO:teuthology.orchestra.run.smithi032.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 1.7 MB/s | 90 kB 00:00 2024-03-25T17:15:17.300 INFO:teuthology.orchestra.run.smithi032.stdout:(12/13): python3-kubernetes-26.1.0-2.el9.noarch 3.5 MB/s | 1.0 MB 00:00 2024-03-25T17:15:17.367 INFO:teuthology.orchestra.run.smithi032.stdout:(13/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el 72 kB/s | 52 kB 00:00 2024-03-25T17:15:17.368 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:17.369 INFO:teuthology.orchestra.run.smithi032.stdout:Total 2.5 MB/s | 2.0 MB 00:00 2024-03-25T17:15:17.369 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:15:17.384 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:15:17.384 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:15:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:15:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:15:17.592 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:18 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:17.741 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:17.742 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 85 k 2024-03-25T17:15:17.743 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 231 k 2024-03-25T17:15:17.743 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:15:18.016 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:15:18.181 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T17:15:18.237 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T17:15:18.291 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T17:15:18.336 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T17:15:18.378 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T17:15:18.476 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T17:15:18.585 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T17:15:18.638 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T17:15:18.987 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 69 kB/s | 85 kB 00:01 2024-03-25T17:15:18.988 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:18.989 INFO:teuthology.orchestra.run.smithi136.stdout:Total 68 kB/s | 85 kB 00:01 2024-03-25T17:15:18.989 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:15:18.997 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:15:18.997 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:15:19.058 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:15:19.058 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:15:19.197 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:15:19.294 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:19.407 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T17:15:19.450 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T17:15:19.484 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T17:15:19.533 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T17:15:19.562 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:15:19.702 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:20.014 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:20.014 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:20.014 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:15:20.014 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:20.014 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:20.014 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:20.274 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror 2024-03-25T17:15:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T17:15:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T17:15:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T17:15:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T17:15:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T17:15:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T17:15:20.301 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T17:15:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T17:15:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T17:15:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T17:15:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T17:15:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T17:15:20.302 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T17:15:20.671 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T17:15:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T17:15:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T17:15:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T17:15:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T17:15:20.672 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:20.672 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:20.844 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:21 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:20.914 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T17:15:20.997 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:20.997 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:20.997 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:15:20.998 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:20.998 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:15:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:15:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:20.998 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:15:20.998 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:20.998 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:15:20.998 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:20.999 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 3.1 M 2024-03-25T17:15:20.999 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 13 M 2024-03-25T17:15:20.999 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:15:21.488 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:10 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:21.645 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:21.646 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:15:21.647 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:15:21.647 INFO:teuthology.orchestra.run.smithi032.stdout:Install 6 Packages 2024-03-25T17:15:21.647 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:21.647 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 6.8 M 2024-03-25T17:15:21.647 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 33 M 2024-03-25T17:15:21.647 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:15:22.185 INFO:teuthology.orchestra.run.smithi032.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 21 MB/s | 5.8 MB 00:00 2024-03-25T17:15:22.210 INFO:teuthology.orchestra.run.smithi032.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 809 kB/s | 248 kB 00:00 2024-03-25T17:15:22.593 INFO:teuthology.orchestra.run.smithi032.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 151 kB/s | 58 kB 00:00 2024-03-25T17:15:22.719 INFO:teuthology.orchestra.run.smithi032.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 407 kB/s | 51 kB 00:00 2024-03-25T17:15:22.786 INFO:teuthology.orchestra.run.smithi032.stdout:(5/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 914 kB/s | 548 kB 00:00 2024-03-25T17:15:23.044 INFO:teuthology.orchestra.run.smithi032.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 121 kB/s | 138 kB 00:01 2024-03-25T17:15:23.047 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:23.047 INFO:teuthology.orchestra.run.smithi032.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-03-25T17:15:23.061 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:15:23.073 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:15:23.073 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:15:23.166 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:15:23.167 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:15:23.466 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:15:23.623 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T17:15:23.688 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T17:15:24.246 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T17:15:24.325 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T17:15:24.387 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T17:15:24.416 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:15:25.269 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T17:15:25.269 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T17:15:25.269 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T17:15:25.269 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T17:15:25.269 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T17:15:25.269 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T17:15:25.678 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:25.679 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:26.044 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse 2024-03-25T17:15:26.621 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:15 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:26.777 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:26.777 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:26.777 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:15:26.777 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:26.777 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:26.778 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 964 k 2024-03-25T17:15:26.779 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 2.8 M 2024-03-25T17:15:26.779 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:15:26.895 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 676 kB/s | 79 kB 00:00 2024-03-25T17:15:31.495 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 188 kB/s | 885 kB 00:04 2024-03-25T17:15:31.497 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:31.497 INFO:teuthology.orchestra.run.smithi032.stdout:Total 204 kB/s | 964 kB 00:04 2024-03-25T17:15:31.497 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:15:31.522 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:15:31.522 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:15:31.602 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:15:31.603 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:15:31.787 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:15:31.925 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T17:15:31.998 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:15:32.439 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 276 kB/s | 3.1 MB 00:11 2024-03-25T17:15:32.440 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:32.440 INFO:teuthology.orchestra.run.smithi136.stdout:Total 276 kB/s | 3.1 MB 00:11 2024-03-25T17:15:32.440 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:15:32.452 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:15:32.452 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:15:32.528 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:15:32.529 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:15:32.656 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:15:32.656 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:15:32.951 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:15:32.993 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T17:15:32.993 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:32.994 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:15:32.994 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T17:15:32.994 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:32.994 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:33.014 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:33.039 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:33.039 INFO:teuthology.orchestra.run.smithi136.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:15:33.040 INFO:teuthology.orchestra.run.smithi136.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T17:15:33.040 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:15:33.040 INFO:teuthology.orchestra.run.smithi136.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:15:33.040 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:33.286 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-volume 2024-03-25T17:15:33.869 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:22 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:34.026 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:34.026 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:34.026 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-03-25T17:15:34.027 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:34.028 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 332 k 2024-03-25T17:15:34.028 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 1.5 M 2024-03-25T17:15:34.028 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:15:34.141 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 611 kB/s | 69 kB 00:00 2024-03-25T17:15:34.212 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:34.212 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:34.212 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:15:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:34.213 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:34.213 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:34.448 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd 2024-03-25T17:15:35.030 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:36 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:35.178 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout: Package Architecture Version Repository Size 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:15:35.179 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:35.180 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 171 k 2024-03-25T17:15:35.180 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 494 k 2024-03-25T17:15:35.180 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:15:35.408 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 191 kB/s | 264 kB 00:01 2024-03-25T17:15:35.409 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:35.410 INFO:teuthology.orchestra.run.smithi032.stdout:Total 240 kB/s | 332 kB 00:01 2024-03-25T17:15:35.410 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:15:35.420 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:15:35.420 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:15:35.461 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:15:35.462 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:15:35.617 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:15:35.813 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T17:15:35.843 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:15:35.859 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T17:15:35.859 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:15:35.859 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T17:15:35.859 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:36.375 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 143 kB/s | 171 kB 00:01 2024-03-25T17:15:36.376 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:36.376 INFO:teuthology.orchestra.run.smithi136.stdout:Total 143 kB/s | 171 kB 00:01 2024-03-25T17:15:36.377 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:15:36.386 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:15:36.386 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:15:36.445 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:15:36.445 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:15:36.568 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T17:15:36.590 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:15:36.707 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:36.925 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T17:15:36.925 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:36.925 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:15:36.926 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T17:15:36.926 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T17:15:36.926 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:36.926 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:37.148 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:37.258 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel 2024-03-25T17:15:37.450 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:37.450 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:37.450 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:15:37.450 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:37.450 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:37.450 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:15:37.693 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:15:37.839 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:37.994 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:37.995 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:37.995 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:15:37.995 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:37.995 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:15:37.995 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T17:15:37.995 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:37.995 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:15:37.995 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:37.996 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:15:37.996 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:37.996 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 127 k 2024-03-25T17:15:37.996 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 456 k 2024-03-25T17:15:37.996 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:15:38.262 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:06:39 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:15:38.414 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout:Installing dependencies: 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:15:38.415 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:15:38.416 INFO:teuthology.orchestra.run.smithi136.stdout:Install 2 Packages 2024-03-25T17:15:38.416 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:15:38.416 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 91 M 2024-03-25T17:15:38.416 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 498 M 2024-03-25T17:15:38.416 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:15:39.266 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 100 kB/s | 127 kB 00:01 2024-03-25T17:15:39.266 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:39.266 INFO:teuthology.orchestra.run.smithi032.stdout:Total 100 kB/s | 127 kB 00:01 2024-03-25T17:15:39.266 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:15:39.273 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:15:39.274 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:15:39.337 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:15:39.337 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:15:39.488 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:15:39.601 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:40.040 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:40.333 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:15:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:40.333 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:40.333 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:40.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2 2024-03-25T17:15:41.173 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:41.274 INFO:teuthology.orchestra.run.smithi032.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:41.331 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:41.332 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:15:41.332 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:41.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs-devel 2024-03-25T17:15:41.968 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:30 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:42.124 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:42.124 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:42.124 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 31 k 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 140 k 2024-03-25T17:15:42.125 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:15:42.668 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 57 kB/s | 31 kB 00:00 2024-03-25T17:15:42.668 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:42.669 INFO:teuthology.orchestra.run.smithi032.stdout:Total 57 kB/s | 31 kB 00:00 2024-03-25T17:15:42.669 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:15:42.672 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:15:42.672 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:15:42.704 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:15:42.704 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:15:42.826 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:15:42.919 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:43.287 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:43.550 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:43.550 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:43.550 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:15:43.550 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:43.550 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:43.550 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:43.711 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2 2024-03-25T17:15:44.291 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:33 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:44.390 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:44.444 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:44.446 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:15:44.446 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:44.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1 2024-03-25T17:15:45.095 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:34 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:45.198 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:45.253 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:45.254 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:15:45.254 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:45.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados 2024-03-25T17:15:45.886 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:34 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:45.985 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:46.039 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:46.041 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:15:46.041 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:46.090 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw 2024-03-25T17:15:46.662 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:35 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:46.761 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:46.815 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:46.816 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:15:46.816 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:46.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs 2024-03-25T17:15:47.443 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:36 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:47.542 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:47.599 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:47.600 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:15:47.600 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:47.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd 2024-03-25T17:15:48.229 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:37 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:48.328 INFO:teuthology.orchestra.run.smithi032.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:15:48.383 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:48.384 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T17:15:48.384 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:48.432 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse 2024-03-25T17:15:49.010 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:38 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:49.165 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:49.166 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:49.166 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-03-25T17:15:49.166 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:49.166 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 85 k 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 231 k 2024-03-25T17:15:49.167 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:15:50.121 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 89 kB/s | 85 kB 00:00 2024-03-25T17:15:50.121 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:15:50.122 INFO:teuthology.orchestra.run.smithi032.stdout:Total 89 kB/s | 85 kB 00:00 2024-03-25T17:15:50.122 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:15:50.131 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:15:50.131 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:15:50.194 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:15:50.194 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:15:50.342 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:15:50.455 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:50.852 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:51.157 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:15:51.157 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:15:51.157 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:15:51.157 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:15:51.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror 2024-03-25T17:15:51.976 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:40 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:15:52.130 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:15:52.131 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:15:52.131 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:52.131 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:15:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T17:15:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:52.131 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:15:52.131 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:15:52.132 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:15:52.132 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:15:52.132 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 3.1 M 2024-03-25T17:15:52.132 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 13 M 2024-03-25T17:15:52.132 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:16:03.598 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 276 kB/s | 3.1 MB 00:11 2024-03-25T17:16:03.599 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:16:03.599 INFO:teuthology.orchestra.run.smithi032.stdout:Total 276 kB/s | 3.1 MB 00:11 2024-03-25T17:16:03.599 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:16:03.611 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:16:03.611 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:16:03.692 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:16:03.692 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:16:04.141 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:16:04.198 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:04.223 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:04.223 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T17:16:04.223 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T17:16:04.223 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:16:04.223 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T17:16:04.223 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:05.339 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:05.340 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:05.340 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:16:05.340 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:05.340 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:05.340 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:16:05.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd 2024-03-25T17:16:06.174 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:55 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:16:06.329 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:16:06.330 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:16:06.330 INFO:teuthology.orchestra.run.smithi032.stdout: Package Architecture Version Repository Size 2024-03-25T17:16:06.330 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:16:06.330 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:16:06.330 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T17:16:06.330 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:06.331 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:16:06.331 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:16:06.331 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:16:06.331 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:06.331 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 171 k 2024-03-25T17:16:06.331 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 494 k 2024-03-25T17:16:06.331 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:16:07.209 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 195 kB/s | 171 kB 00:00 2024-03-25T17:16:07.209 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:16:07.210 INFO:teuthology.orchestra.run.smithi032.stdout:Total 195 kB/s | 171 kB 00:00 2024-03-25T17:16:07.210 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:16:07.219 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:16:07.219 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:16:07.281 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:16:07.281 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:16:07.434 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:16:07.536 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:07.934 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:08.221 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:16:08.222 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:08.222 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:16:08.222 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:16:08.222 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:08.222 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:16:08.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:16:09.036 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:06:58 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:16:09.192 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:16:09.193 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:16:09.193 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:16:09.193 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:16:09.193 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:16:09.193 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T17:16:09.194 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-03-25T17:16:09.194 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T17:16:09.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:09.194 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:16:09.194 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:16:09.194 INFO:teuthology.orchestra.run.smithi032.stdout:Install 2 Packages 2024-03-25T17:16:09.194 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:16:09.194 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 91 M 2024-03-25T17:16:09.195 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 498 M 2024-03-25T17:16:09.195 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:16:39.251 INFO:teuthology.orchestra.run.smithi136.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 378 kB/s | 22 MB 01:00 2024-03-25T17:17:26.412 INFO:teuthology.orchestra.run.smithi032.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 298 kB/s | 22 MB 01:17 2024-03-25T17:19:46.515 INFO:teuthology.orchestra.run.smithi032.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 321 kB/s | 68 MB 03:37 2024-03-25T17:19:46.536 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:19:46.536 INFO:teuthology.orchestra.run.smithi032.stdout:Total 427 kB/s | 91 MB 03:37 2024-03-25T17:19:46.536 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:19:46.536 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:19:46.536 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:19:47.142 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:19:47.142 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:19:50.412 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:19:57.922 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:19:58.000 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:20:01.457 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:20:01.458 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T17:20:01.725 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:20:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:20:01.726 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:20:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:20:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:20:01.726 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:20:01.726 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:20:01.940 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-common-debuginfo 2024-03-25T17:20:02.514 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:10:51 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:20:02.668 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:20:02.669 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:20:02.670 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:20:02.670 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 416 M 2024-03-25T17:20:02.670 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 1.7 G 2024-03-25T17:20:02.670 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:20:25.858 INFO:teuthology.orchestra.run.smithi136.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 243 kB/s | 68 MB 04:47 2024-03-25T17:20:25.859 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:20:25.859 INFO:teuthology.orchestra.run.smithi136.stdout:Total 323 kB/s | 91 MB 04:47 2024-03-25T17:20:25.860 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:20:25.864 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:20:25.864 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:20:26.448 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:20:26.448 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:20:29.543 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:20:36.665 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T17:20:36.740 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:20:40.085 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T17:20:40.085 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T17:20:40.359 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T17:20:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:20:40.360 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:20:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:20:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:20:40.360 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:20:40.360 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:20:40.580 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-common-debuginfo 2024-03-25T17:20:41.157 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:11:42 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:20:41.318 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:20:41.319 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:20:41.320 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:20:41.320 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 416 M 2024-03-25T17:20:41.320 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 1.7 G 2024-03-25T17:20:41.320 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:35:23.838 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 463 kB/s | 416 MB 15:21 2024-03-25T17:35:23.860 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:35:23.860 INFO:teuthology.orchestra.run.smithi032.stdout:Total 463 kB/s | 416 MB 15:21 2024-03-25T17:35:23.860 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:35:23.860 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:35:23.860 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:35:26.430 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:35:26.430 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:36:11.114 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:36:11.202 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:36:19.368 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:36:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:36:19.632 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:36:19.632 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:36:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:36:19.633 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:36:19.633 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:36:19.981 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:36:20.557 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:27:09 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:36:20.712 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:36:20.712 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:36:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:36:20.712 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:36:20.712 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:36:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:36:20.712 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:36:20.712 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:36:20.713 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================== 2024-03-25T17:36:20.713 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:36:20.713 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:36:20.714 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 1.4 M 2024-03-25T17:36:20.714 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 6.4 M 2024-03-25T17:36:20.714 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:36:27.727 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 208 kB/s | 1.4 MB 00:07 2024-03-25T17:36:27.728 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:36:27.729 INFO:teuthology.orchestra.run.smithi032.stdout:Total 208 kB/s | 1.4 MB 00:07 2024-03-25T17:36:27.729 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:36:27.733 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:36:27.733 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:36:27.779 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:36:27.779 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:36:28.052 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:36:28.161 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:36:28.589 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:36:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:36:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:36:28.859 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:36:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:36:28.859 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:36:28.859 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:36:29.042 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:36:29.614 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:27:18 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:36:29.769 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 176 M 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 753 M 2024-03-25T17:36:29.770 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:40:57.638 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 350 kB/s | 416 MB 20:16 2024-03-25T17:40:57.639 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:40:57.639 INFO:teuthology.orchestra.run.smithi136.stdout:Total 350 kB/s | 416 MB 20:16 2024-03-25T17:40:57.639 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:40:57.642 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:40:57.642 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:41:00.126 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:41:00.127 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:41:42.336 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:41:42.428 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:41:49.588 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 562 kB/s | 176 MB 05:19 2024-03-25T17:41:49.589 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:49.589 INFO:teuthology.orchestra.run.smithi032.stdout:Total 562 kB/s | 176 MB 05:19 2024-03-25T17:41:49.589 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T17:41:49.592 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T17:41:49.593 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T17:41:50.521 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:41:50.701 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T17:41:50.701 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T17:41:50.771 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:41:50.771 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:41:50.771 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:41:50.771 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:41:50.771 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:41:50.771 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:41:51.009 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:41:51.577 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:32:52 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:41:51.725 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:41:51.726 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:41:51.726 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:41:51.726 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:41:51.726 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:41:51.726 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:41:51.726 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:41:51.726 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:41:51.726 INFO:teuthology.orchestra.run.smithi136.stdout:======================================================================================== 2024-03-25T17:41:51.727 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:41:51.727 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:41:51.727 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 1.4 M 2024-03-25T17:41:51.727 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 6.4 M 2024-03-25T17:41:51.727 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:41:55.557 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 381 kB/s | 1.4 MB 00:03 2024-03-25T17:41:55.558 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:55.558 INFO:teuthology.orchestra.run.smithi136.stdout:Total 381 kB/s | 1.4 MB 00:03 2024-03-25T17:41:55.558 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:41:55.562 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:41:55.562 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:41:55.608 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:41:55.608 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:41:55.861 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:41:55.971 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:41:56.384 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:41:56.696 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:41:56.696 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:41:56.696 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:41:56.696 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:41:56.696 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:41:56.696 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:41:56.909 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:41:57.473 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:32:58 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:41:57.621 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:41:57.621 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 176 M 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 753 M 2024-03-25T17:41:57.622 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T17:42:09.709 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T17:42:09.804 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:42:14.556 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:42:14.846 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:42:14.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:42:14.847 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T17:42:14.847 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:42:14.847 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:42:14.847 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T17:42:15.170 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:42:15.746 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 0:33:04 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T17:42:15.901 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T17:42:15.902 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T17:42:15.903 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 672 M 2024-03-25T17:42:15.903 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 3.1 G 2024-03-25T17:42:15.903 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T17:50:42.832 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 342 kB/s | 176 MB 08:45 2024-03-25T17:50:42.832 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:50:42.832 INFO:teuthology.orchestra.run.smithi136.stdout:Total 342 kB/s | 176 MB 08:45 2024-03-25T17:50:42.832 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T17:50:42.835 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T17:50:42.835 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T17:50:43.891 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T17:50:43.892 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T17:51:01.811 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T17:51:01.917 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:51:06.708 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:51:06.970 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:51:06.970 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:51:06.971 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T17:51:06.971 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:51:06.971 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:51:06.971 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T17:51:07.197 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:51:07.756 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 0:42:08 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T17:51:07.904 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T17:51:07.905 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:51:07.905 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 672 M 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 3.1 G 2024-03-25T17:51:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:13:44.588 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 507 kB/s | 672 MB 22:36 2024-03-25T18:13:44.588 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:44.588 INFO:teuthology.orchestra.run.smithi136.stdout:Total 507 kB/s | 672 MB 22:36 2024-03-25T18:13:44.588 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:13:44.592 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:13:44.592 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:13:48.561 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:13:48.561 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:14:59.704 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:14:59.866 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:15:09.748 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:15:10.011 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:15:10.012 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:15:10.012 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:15:10.012 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:15:10.012 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:15:10.012 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:15:10.297 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:15:10.866 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:06:11 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:15:10.964 INFO:teuthology.orchestra.run.smithi136.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:15:11.017 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:15:11.018 INFO:teuthology.orchestra.run.smithi136.stdout:Nothing to do. 2024-03-25T18:15:11.018 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:15:11.083 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:15:11.649 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:06:12 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:15:11.796 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:15:11.797 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:15:11.797 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:15:11.797 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:15:11.797 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:15:11.797 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:15:11.797 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:15:11.797 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:15:11.798 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:15:11.798 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:15:11.798 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:15:11.798 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 20 M 2024-03-25T18:15:11.798 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 103 M 2024-03-25T18:15:11.798 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:15:59.302 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 425 kB/s | 20 MB 00:47 2024-03-25T18:15:59.303 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:15:59.303 INFO:teuthology.orchestra.run.smithi136.stdout:Total 425 kB/s | 20 MB 00:47 2024-03-25T18:15:59.303 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:15:59.307 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:15:59.307 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:15:59.457 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:15:59.457 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:16:01.692 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:16:01.867 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:02.875 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:03.135 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:16:03.136 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:16:03.136 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:16:03.136 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:16:03.136 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:16:03.136 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:16:03.316 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:16:03.889 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:07:04 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:16:04.046 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:16:04.047 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:16:04.047 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:16:04.047 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:16:04.047 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:16:04.047 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:16:04.047 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:16:04.047 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:16:04.048 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:16:04.048 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:16:04.048 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:16:04.048 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 48 M 2024-03-25T18:16:04.048 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 173 M 2024-03-25T18:16:04.048 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:16:34.250 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 334 kB/s | 672 MB 34:18 2024-03-25T18:16:34.250 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:16:34.250 INFO:teuthology.orchestra.run.smithi032.stdout:Total 334 kB/s | 672 MB 34:18 2024-03-25T18:16:34.251 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:16:34.256 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:16:34.256 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:16:38.411 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:16:38.411 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:17:53.412 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:17:53.521 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:01.227 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:18:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:18:01.506 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:18:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:01.506 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:18:01.506 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:18:01.794 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-base-debuginfo 2024-03-25T18:18:02.370 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:08:51 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:18:02.469 INFO:teuthology.orchestra.run.smithi032.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T18:18:02.524 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:18:02.525 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-03-25T18:18:02.526 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:18:02.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T18:18:03.169 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:08:52 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:18:03.323 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:18:03.324 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:18:03.324 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:18:03.324 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:18:03.324 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 20 M 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 103 M 2024-03-25T18:18:03.325 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:18:20.653 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 358 kB/s | 48 MB 02:16 2024-03-25T18:18:20.654 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:20.654 INFO:teuthology.orchestra.run.smithi136.stdout:Total 358 kB/s | 48 MB 02:16 2024-03-25T18:18:20.654 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:18:20.658 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:18:20.658 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:18:20.970 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:18:20.970 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:18:25.695 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:18:25.776 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:27.088 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:18:27.361 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:18:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:27.361 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:18:27.361 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:18:27.550 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:18:28.121 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:09:29 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:18:28.268 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:18:28.269 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:18:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:18:28.269 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:18:28.269 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:18:28.269 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:18:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:18:28.270 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:18:28.270 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:18:28.270 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:18:28.270 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:18:28.270 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 99 M 2024-03-25T18:18:28.270 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 449 M 2024-03-25T18:18:28.270 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:18:33.686 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 664 kB/s | 20 MB 00:30 2024-03-25T18:18:33.686 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:18:33.687 INFO:teuthology.orchestra.run.smithi032.stdout:Total 664 kB/s | 20 MB 00:30 2024-03-25T18:18:33.687 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:18:33.691 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:18:33.691 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:18:33.849 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:18:33.850 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:18:36.194 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:18:36.294 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:37.370 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:37.642 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:18:37.642 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:18:37.642 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:18:37.642 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:18:37.642 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:18:37.642 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:18:37.924 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T18:18:38.499 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:09:27 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:18:38.655 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:18:38.655 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:18:38.655 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:18:38.656 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 48 M 2024-03-25T18:18:38.657 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 173 M 2024-03-25T18:18:38.657 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:20:18.729 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 488 kB/s | 48 MB 01:40 2024-03-25T18:20:18.730 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:20:18.730 INFO:teuthology.orchestra.run.smithi032.stdout:Total 488 kB/s | 48 MB 01:40 2024-03-25T18:20:18.730 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:20:18.733 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:20:18.733 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:20:19.060 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:20:19.060 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:20:24.076 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:20:24.170 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:20:25.545 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:20:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:20:25.811 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:20:25.811 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:20:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:20:25.812 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:20:25.812 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:20:25.993 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T18:20:26.571 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:11:15 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:20:26.726 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 99 M 2024-03-25T18:20:26.727 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 449 M 2024-03-25T18:20:26.728 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:21:21.838 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 581 kB/s | 99 MB 02:53 2024-03-25T18:21:21.839 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:21:21.839 INFO:teuthology.orchestra.run.smithi136.stdout:Total 581 kB/s | 99 MB 02:53 2024-03-25T18:21:21.839 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:21:21.842 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:21:21.842 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:21:22.451 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:21:22.451 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:21:32.753 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:21:32.860 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:35.768 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:36.026 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:21:36.026 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:21:36.026 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:21:36.026 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:21:36.026 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:21:36.026 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:21:36.225 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:21:36.794 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:12:37 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:21:36.943 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 287 M 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 1.2 G 2024-03-25T18:21:36.944 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:23:50.193 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 496 kB/s | 99 MB 03:23 2024-03-25T18:23:50.193 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:23:50.193 INFO:teuthology.orchestra.run.smithi032.stdout:Total 496 kB/s | 99 MB 03:23 2024-03-25T18:23:50.193 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:23:50.196 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:23:50.196 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:23:50.834 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:23:50.834 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:24:01.838 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:24:01.934 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:24:04.740 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:24:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:24:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:24:05.010 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:24:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:24:05.010 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:24:05.010 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:24:05.197 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T18:24:05.773 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:14:54 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:24:05.927 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:24:05.928 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:24:05.929 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:24:05.929 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 287 M 2024-03-25T18:24:05.929 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 1.2 G 2024-03-25T18:24:05.929 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:26:53.869 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 927 kB/s | 287 MB 05:16 2024-03-25T18:26:53.869 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:26:53.870 INFO:teuthology.orchestra.run.smithi136.stdout:Total 927 kB/s | 287 MB 05:16 2024-03-25T18:26:53.870 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:26:53.873 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:26:53.873 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:26:55.578 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:26:55.578 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:27:24.990 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:27:25.522 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:27:29.971 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:27:30.260 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:27:30.260 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:30.260 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:27:30.260 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:27:30.260 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:30.261 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:27:30.479 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:27:31.052 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:18:32 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:27:31.200 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:31.201 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 5.9 M 2024-03-25T18:27:31.202 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 28 M 2024-03-25T18:27:31.202 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:27:38.550 INFO:teuthology.orchestra.run.smithi136.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 828 kB/s | 5.9 MB 00:07 2024-03-25T18:27:38.551 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:27:38.551 INFO:teuthology.orchestra.run.smithi136.stdout:Total 828 kB/s | 5.9 MB 00:07 2024-03-25T18:27:38.552 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:27:38.556 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:27:38.556 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:27:38.627 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:27:38.628 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:27:39.369 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:27:39.490 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:27:40.076 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:27:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:27:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:40.339 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:27:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:27:40.339 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:40.339 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:27:40.560 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:27:41.126 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:18:42 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:27:41.276 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:27:41.276 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:27:41.276 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:41.276 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:27:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:27:41.276 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:41.276 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:27:41.277 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:41.277 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:27:41.277 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:41.277 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 109 k 2024-03-25T18:27:41.277 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 440 k 2024-03-25T18:27:41.277 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:27:42.178 INFO:teuthology.orchestra.run.smithi136.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 121 kB/s | 109 kB 00:00 2024-03-25T18:27:42.178 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:27:42.179 INFO:teuthology.orchestra.run.smithi136.stdout:Total 121 kB/s | 109 kB 00:00 2024-03-25T18:27:42.179 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:27:42.182 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:27:42.182 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:27:42.220 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:27:42.220 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:27:42.338 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:27:42.444 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:27:42.819 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:27:43.096 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:27:43.096 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:43.096 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:27:43.096 INFO:teuthology.orchestra.run.smithi136.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:27:43.097 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:43.097 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:27:43.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:27:43.835 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:18:44 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:27:43.984 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:27:43.984 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:43.984 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:27:43.984 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 5.4 M 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 25 M 2024-03-25T18:27:43.985 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:27:55.270 INFO:teuthology.orchestra.run.smithi136.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 486 kB/s | 5.4 MB 00:11 2024-03-25T18:27:55.271 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:27:55.271 INFO:teuthology.orchestra.run.smithi136.stdout:Total 486 kB/s | 5.4 MB 00:11 2024-03-25T18:27:55.272 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:27:55.276 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:27:55.276 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:27:55.345 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:27:55.345 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:27:56.017 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:27:56.127 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:27:56.674 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:27:56.959 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:27:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:56.960 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:27:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:27:56.960 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:56.960 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:27:57.153 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librados2-debuginfo 2024-03-25T18:27:57.718 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:18:58 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:27:57.868 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:27:57.869 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:57.869 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:27:57.869 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:57.869 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:27:57.869 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:27:57.869 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:57.869 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:27:57.869 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:27:57.870 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:27:57.870 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:27:57.870 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 44 M 2024-03-25T18:27:57.870 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 253 M 2024-03-25T18:27:57.870 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:28:55.579 INFO:teuthology.orchestra.run.smithi136.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 786 kB/s | 44 MB 00:57 2024-03-25T18:28:55.579 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:28:55.579 INFO:teuthology.orchestra.run.smithi136.stdout:Total 786 kB/s | 44 MB 00:57 2024-03-25T18:28:55.580 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:28:55.582 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:28:55.582 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:28:55.881 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:28:55.882 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:29:00.934 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:29:01.026 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:29:02.936 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:29:03.203 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:29:03.203 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:29:03.203 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:29:03.203 INFO:teuthology.orchestra.run.smithi136.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:29:03.203 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:29:03.203 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:29:03.399 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:29:03.966 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:20:04 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:29:04.117 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:29:04.117 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 52 M 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 308 M 2024-03-25T18:29:04.118 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 560 kB/s | 52 MB 01:34 2024-03-25T18:30:38.963 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout:Total 560 kB/s | 52 MB 01:34 2024-03-25T18:30:38.964 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:30:38.967 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:30:39.311 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:30:39.311 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:30:45.319 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:30:45.510 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:30:45.800 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 734 kB/s | 287 MB 06:39 2024-03-25T18:30:45.801 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:45.801 INFO:teuthology.orchestra.run.smithi032.stdout:Total 734 kB/s | 287 MB 06:39 2024-03-25T18:30:45.801 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:30:45.805 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:30:45.805 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:30:47.594 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:30:47.595 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:30:47.738 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:30:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:30:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:47.997 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:30:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:47.997 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:47.997 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:30:48.203 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:30:48.765 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:21:49 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:30:48.915 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:30:48.915 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:48.915 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:30:48.915 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:48.915 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:30:48.915 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:30:48.915 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:48.915 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:30:48.916 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:48.916 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:30:48.916 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:48.916 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 615 k 2024-03-25T18:30:48.916 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.1 M 2024-03-25T18:30:48.916 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:30:50.055 INFO:teuthology.orchestra.run.smithi136.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 540 kB/s | 615 kB 00:01 2024-03-25T18:30:50.056 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:50.056 INFO:teuthology.orchestra.run.smithi136.stdout:Total 539 kB/s | 615 kB 00:01 2024-03-25T18:30:50.056 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:30:50.061 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:30:50.061 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:30:50.102 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:30:50.102 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:30:50.271 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:30:50.371 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:30:50.779 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:30:51.047 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:30:51.047 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:51.047 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:30:51.047 INFO:teuthology.orchestra.run.smithi136.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:51.047 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:51.048 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:30:51.238 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:30:51.807 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:21:52 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:30:51.958 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:30:51.959 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:51.959 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:30:51.959 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:51.959 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:30:51.959 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:30:51.960 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:51.960 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:30:51.960 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:51.960 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:30:51.960 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:51.960 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 771 k 2024-03-25T18:30:51.960 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.7 M 2024-03-25T18:30:51.960 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:30:53.616 INFO:teuthology.orchestra.run.smithi136.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 466 kB/s | 771 kB 00:01 2024-03-25T18:30:53.617 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:53.617 INFO:teuthology.orchestra.run.smithi136.stdout:Total 465 kB/s | 771 kB 00:01 2024-03-25T18:30:53.617 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:30:53.621 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:30:53.621 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:30:53.662 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:30:53.662 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:30:53.846 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:30:53.948 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:30:54.311 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:30:54.580 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:30:54.580 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:54.580 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:30:54.580 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:54.580 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:54.580 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:30:54.800 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:30:55.366 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:21:56 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:30:55.519 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:30:55.519 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:55.519 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 1.1 M 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 4.3 M 2024-03-25T18:30:55.520 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:30:58.601 INFO:teuthology.orchestra.run.smithi136.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 374 kB/s | 1.1 MB 00:03 2024-03-25T18:30:58.602 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:30:58.603 INFO:teuthology.orchestra.run.smithi136.stdout:Total 373 kB/s | 1.1 MB 00:03 2024-03-25T18:30:58.603 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:30:58.606 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:30:58.607 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:30:58.651 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:30:58.651 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:30:58.892 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:30:59.000 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:30:59.391 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:30:59.669 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:30:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:59.670 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:30:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:30:59.670 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:30:59.670 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:30:59.892 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:31:00.462 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:22:01 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:31:00.613 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:31:00.614 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:00.614 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:31:00.614 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:00.614 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:31:00.614 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:31:00.614 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:00.614 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:31:00.614 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:00.615 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:31:00.615 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:00.615 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 337 k 2024-03-25T18:31:00.615 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 1.0 M 2024-03-25T18:31:00.615 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:31:02.304 INFO:teuthology.orchestra.run.smithi136.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 200 kB/s | 337 kB 00:01 2024-03-25T18:31:02.304 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:02.304 INFO:teuthology.orchestra.run.smithi136.stdout:Total 200 kB/s | 337 kB 00:01 2024-03-25T18:31:02.305 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:31:02.309 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:31:02.309 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:31:02.349 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:31:02.349 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:31:02.494 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:31:02.611 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:31:02.965 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:31:03.227 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:31:03.227 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:03.227 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:31:03.227 INFO:teuthology.orchestra.run.smithi136.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:03.227 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:03.228 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:31:03.455 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:31:04.015 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:22:05 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:31:04.165 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:31:04.166 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:04.166 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:31:04.166 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:04.166 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:31:04.166 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:31:04.166 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:04.167 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:31:04.167 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:04.167 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:31:04.167 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:04.167 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 509 k 2024-03-25T18:31:04.167 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 2.5 M 2024-03-25T18:31:04.167 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:31:06.132 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 259 kB/s | 509 kB 00:01 2024-03-25T18:31:06.132 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:06.133 INFO:teuthology.orchestra.run.smithi136.stdout:Total 259 kB/s | 509 kB 00:01 2024-03-25T18:31:06.133 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:31:06.137 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:31:06.137 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:31:06.177 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:31:06.177 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:31:06.339 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:31:06.447 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:31:06.839 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:31:07.101 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:31:07.101 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:07.102 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:31:07.102 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:07.102 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:07.102 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:31:07.339 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:31:07.906 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:22:08 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:31:08.057 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:31:08.057 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:08.057 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 52 M 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 298 M 2024-03-25T18:31:08.058 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:31:18.650 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:31:18.758 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:31:19.994 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:31:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:31:20.278 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:20.279 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:31:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:20.279 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:20.279 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:31:20.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:31:21.061 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:22:10 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:31:21.216 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:31:21.217 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 5.9 M 2024-03-25T18:31:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 28 M 2024-03-25T18:31:21.218 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:31:34.441 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 460 kB/s | 5.9 MB 00:13 2024-03-25T18:31:34.442 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:34.442 INFO:teuthology.orchestra.run.smithi032.stdout:Total 460 kB/s | 5.9 MB 00:13 2024-03-25T18:31:34.443 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:31:34.447 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:31:34.447 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:31:34.522 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:31:34.522 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:31:35.310 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:31:35.401 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:31:35.952 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:31:36.251 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:31:36.251 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:36.251 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:31:36.251 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:36.251 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:36.251 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:31:36.430 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:31:37.006 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:22:26 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:31:37.162 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:31:37.162 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:37.162 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:31:37.162 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:37.162 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:31:37.162 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:31:37.162 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:37.162 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:31:37.163 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:37.163 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:31:37.163 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:37.163 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 109 k 2024-03-25T18:31:37.163 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 440 k 2024-03-25T18:31:37.163 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:31:37.778 INFO:teuthology.orchestra.run.smithi032.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 177 kB/s | 109 kB 00:00 2024-03-25T18:31:37.778 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:37.779 INFO:teuthology.orchestra.run.smithi032.stdout:Total 177 kB/s | 109 kB 00:00 2024-03-25T18:31:37.779 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:31:37.782 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:31:37.783 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:31:37.819 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:31:37.819 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:31:37.936 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:31:38.037 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:31:38.399 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:31:38.676 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:31:38.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:38.676 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:31:38.676 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:38.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:38.677 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:31:38.863 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:31:39.438 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:22:28 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:31:39.593 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:31:39.593 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:31:39.594 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:39.595 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 5.4 M 2024-03-25T18:31:39.595 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 25 M 2024-03-25T18:31:39.595 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:31:46.780 INFO:teuthology.orchestra.run.smithi032.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 763 kB/s | 5.4 MB 00:07 2024-03-25T18:31:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:31:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:Total 763 kB/s | 5.4 MB 00:07 2024-03-25T18:31:46.781 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:31:46.786 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:31:46.786 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:31:46.858 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:31:46.858 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:31:47.580 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:31:47.699 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:31:48.256 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:31:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:31:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:48.517 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:31:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:31:48.517 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:48.517 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:31:48.723 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librados2-debuginfo 2024-03-25T18:31:49.298 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:22:38 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:31:49.453 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:31:49.454 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:31:49.455 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:31:49.455 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 44 M 2024-03-25T18:31:49.455 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 253 M 2024-03-25T18:31:49.455 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:32:28.169 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 659 kB/s | 52 MB 01:20 2024-03-25T18:32:28.170 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:32:28.170 INFO:teuthology.orchestra.run.smithi136.stdout:Total 659 kB/s | 52 MB 01:20 2024-03-25T18:32:28.171 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:32:28.175 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:32:28.175 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:32:28.511 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:32:28.511 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:32:34.418 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:32:34.809 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:32:36.919 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:32:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:32:37.173 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:32:37.173 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:32:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:37.174 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:32:37.174 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:32:37.391 DEBUG:teuthology.orchestra.run.smithi136:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:32:37.963 INFO:teuthology.orchestra.run.smithi136.stdout:Last metadata expiration check: 1:23:38 ago on Mon 25 Mar 2024 05:08:59 PM UTC. 2024-03-25T18:32:38.114 INFO:teuthology.orchestra.run.smithi136.stdout:Dependencies resolved. 2024-03-25T18:32:38.114 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:32:38.114 INFO:teuthology.orchestra.run.smithi136.stdout: Package Arch Version Repo Size 2024-03-25T18:32:38.115 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:32:38.115 INFO:teuthology.orchestra.run.smithi136.stdout:Installing: 2024-03-25T18:32:38.115 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:32:38.115 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:32:38.115 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction Summary 2024-03-25T18:32:38.115 INFO:teuthology.orchestra.run.smithi136.stdout:================================================================================ 2024-03-25T18:32:38.115 INFO:teuthology.orchestra.run.smithi136.stdout:Install 1 Package 2024-03-25T18:32:38.115 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:32:38.116 INFO:teuthology.orchestra.run.smithi136.stdout:Total download size: 1.3 M 2024-03-25T18:32:38.116 INFO:teuthology.orchestra.run.smithi136.stdout:Installed size: 5.5 M 2024-03-25T18:32:38.116 INFO:teuthology.orchestra.run.smithi136.stdout:Downloading Packages: 2024-03-25T18:32:39.087 INFO:teuthology.orchestra.run.smithi136.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 1.3 MB/s | 1.3 MB 00:00 2024-03-25T18:32:39.087 INFO:teuthology.orchestra.run.smithi136.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:32:39.088 INFO:teuthology.orchestra.run.smithi136.stdout:Total 1.3 MB/s | 1.3 MB 00:00 2024-03-25T18:32:39.088 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction check 2024-03-25T18:32:39.091 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction check succeeded. 2024-03-25T18:32:39.092 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction test 2024-03-25T18:32:39.135 INFO:teuthology.orchestra.run.smithi136.stdout:Transaction test succeeded. 2024-03-25T18:32:39.135 INFO:teuthology.orchestra.run.smithi136.stdout:Running transaction 2024-03-25T18:32:39.379 INFO:teuthology.orchestra.run.smithi136.stdout: Preparing : 1/1 2024-03-25T18:32:39.478 INFO:teuthology.orchestra.run.smithi136.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:32:39.891 INFO:teuthology.orchestra.run.smithi136.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:32:40.157 INFO:teuthology.orchestra.run.smithi136.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:32:40.157 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:32:40.157 INFO:teuthology.orchestra.run.smithi136.stdout:Installed: 2024-03-25T18:32:40.157 INFO:teuthology.orchestra.run.smithi136.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:32:40.157 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:32:40.157 INFO:teuthology.orchestra.run.smithi136.stdout:Complete! 2024-03-25T18:32:40.365 DEBUG:teuthology.parallel:result is None 2024-03-25T18:33:27.265 INFO:teuthology.orchestra.run.smithi032.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 464 kB/s | 44 MB 01:37 2024-03-25T18:33:27.265 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:33:27.265 INFO:teuthology.orchestra.run.smithi032.stdout:Total 464 kB/s | 44 MB 01:37 2024-03-25T18:33:27.266 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:33:27.269 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:33:27.269 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:33:27.582 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:33:27.582 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:33:32.929 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:33:33.107 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:33:34.897 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:33:35.153 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:33:35.153 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:33:35.153 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:33:35.153 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:33:35.153 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:33:35.153 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:33:35.329 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:33:35.904 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:24:24 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:33:36.061 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:33:36.140 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:33:36.140 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:33:36.140 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:33:36.140 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:33:36.140 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:33:36.141 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:33:36.141 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:33:36.141 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:33:36.141 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:33:36.141 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:33:36.141 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 52 M 2024-03-25T18:33:36.141 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 308 M 2024-03-25T18:33:36.141 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:35:22.498 INFO:teuthology.orchestra.run.smithi032.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 499 kB/s | 52 MB 01:46 2024-03-25T18:35:22.499 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:22.499 INFO:teuthology.orchestra.run.smithi032.stdout:Total 499 kB/s | 52 MB 01:46 2024-03-25T18:35:22.499 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:35:22.502 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:35:22.502 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:35:22.863 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:35:22.863 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:35:29.192 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:35:29.299 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:31.452 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:31.722 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:35:31.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:31.722 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:35:31.722 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:31.722 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:31.722 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:35:31.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:35:32.474 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:26:21 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:35:32.628 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:35:32.629 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:32.629 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:35:32.629 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:32.629 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:35:32.629 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:35:32.629 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:32.630 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:35:32.630 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:32.630 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:35:32.630 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:32.630 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 615 k 2024-03-25T18:35:32.630 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 2.1 M 2024-03-25T18:35:32.630 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:35:33.525 INFO:teuthology.orchestra.run.smithi032.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 688 kB/s | 615 kB 00:00 2024-03-25T18:35:33.525 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:33.526 INFO:teuthology.orchestra.run.smithi032.stdout:Total 686 kB/s | 615 kB 00:00 2024-03-25T18:35:33.526 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:35:33.530 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:35:33.530 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:35:33.574 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:35:33.574 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:35:33.752 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:35:33.859 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:35:34.254 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:35:34.511 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:35:34.511 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:34.511 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:35:34.511 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:34.512 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:34.512 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:35:34.713 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:35:35.283 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:26:24 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:35:35.440 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:35:35.440 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:35.440 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:35:35.440 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:35.441 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:35:35.441 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:35:35.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:35.441 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:35:35.441 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:35.441 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:35:35.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:35.442 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 771 k 2024-03-25T18:35:35.442 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 2.7 M 2024-03-25T18:35:35.442 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:35:39.261 INFO:teuthology.orchestra.run.smithi032.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 202 kB/s | 771 kB 00:03 2024-03-25T18:35:39.261 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:39.262 INFO:teuthology.orchestra.run.smithi032.stdout:Total 202 kB/s | 771 kB 00:03 2024-03-25T18:35:39.262 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:35:39.266 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:35:39.266 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:35:39.309 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:35:39.309 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:35:39.542 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:35:39.646 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:35:40.009 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:35:40.280 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:35:40.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:40.280 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:35:40.280 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:40.280 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:40.280 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:35:40.487 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:35:41.059 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:26:30 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:35:41.215 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:41.216 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:35:41.217 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:41.217 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 1.1 M 2024-03-25T18:35:41.217 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 4.3 M 2024-03-25T18:35:41.217 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:35:43.605 INFO:teuthology.orchestra.run.smithi032.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 482 kB/s | 1.1 MB 00:02 2024-03-25T18:35:43.606 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:43.606 INFO:teuthology.orchestra.run.smithi032.stdout:Total 482 kB/s | 1.1 MB 00:02 2024-03-25T18:35:43.607 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:35:43.611 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:35:43.611 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:35:43.656 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:35:43.657 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:35:43.891 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:35:43.990 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:35:44.382 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:35:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:35:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:44.637 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:35:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:44.637 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:44.637 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:35:44.839 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:35:45.414 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:26:34 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:35:45.570 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:35:45.571 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:45.571 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:35:45.571 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:45.571 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:35:45.571 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:35:45.571 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:45.571 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:35:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:35:45.572 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 337 k 2024-03-25T18:35:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 1.0 M 2024-03-25T18:35:45.572 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:35:47.591 INFO:teuthology.orchestra.run.smithi032.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 167 kB/s | 337 kB 00:02 2024-03-25T18:35:47.592 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:47.592 INFO:teuthology.orchestra.run.smithi032.stdout:Total 167 kB/s | 337 kB 00:02 2024-03-25T18:35:47.593 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:35:47.597 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:35:47.597 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:35:47.638 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:35:47.639 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:35:47.788 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:35:47.893 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:35:48.280 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:35:48.544 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:35:48.545 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:48.545 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:35:48.545 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:48.545 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:48.545 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:35:48.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:35:49.325 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:26:38 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:35:49.482 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:35:49.483 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:35:49.483 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:49.483 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:35:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:35:49.483 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:35:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:35:49.484 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 509 k 2024-03-25T18:35:49.484 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 2.5 M 2024-03-25T18:35:49.485 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:35:53.661 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 122 kB/s | 509 kB 00:04 2024-03-25T18:35:53.662 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:35:53.662 INFO:teuthology.orchestra.run.smithi032.stdout:Total 122 kB/s | 509 kB 00:04 2024-03-25T18:35:53.662 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:35:53.666 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:35:53.667 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:35:53.708 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:35:53.708 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:35:53.878 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:35:53.987 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:35:54.376 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:35:54.652 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:35:54.652 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:54.652 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:35:54.652 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:35:54.652 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:54.652 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:35:54.851 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:35:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:26:44 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:35:55.578 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:35:55.579 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:35:55.580 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:35:55.580 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 52 M 2024-03-25T18:35:55.580 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 298 M 2024-03-25T18:35:55.580 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:38:00.727 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 422 kB/s | 52 MB 02:05 2024-03-25T18:38:00.728 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:00.728 INFO:teuthology.orchestra.run.smithi032.stdout:Total 422 kB/s | 52 MB 02:05 2024-03-25T18:38:00.728 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:38:00.731 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:38:00.731 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:38:01.081 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:38:01.081 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:38:07.327 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:38:07.435 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:38:09.436 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:38:09.761 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:38:09.761 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:09.761 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:38:09.761 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:09.761 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:09.761 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:38:09.938 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:38:10.520 INFO:teuthology.orchestra.run.smithi032.stdout:Last metadata expiration check: 1:28:59 ago on Mon 25 Mar 2024 05:09:11 PM UTC. 2024-03-25T18:38:10.674 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-03-25T18:38:10.675 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:38:10.675 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repo Size 2024-03-25T18:38:10.675 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout:================================================================================ 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout:Install 1 Package 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 1.3 M 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout:Installed size: 5.5 M 2024-03-25T18:38:10.676 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-03-25T18:38:12.253 INFO:teuthology.orchestra.run.smithi032.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 813 kB/s | 1.3 MB 00:01 2024-03-25T18:38:12.254 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:38:12.254 INFO:teuthology.orchestra.run.smithi032.stdout:Total 812 kB/s | 1.3 MB 00:01 2024-03-25T18:38:12.254 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-03-25T18:38:12.258 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-03-25T18:38:12.259 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-03-25T18:38:12.305 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-03-25T18:38:12.305 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-03-25T18:38:12.558 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-03-25T18:38:12.671 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:38:13.085 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:38:13.351 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:38:13.351 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:13.351 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-03-25T18:38:13.351 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:38:13.352 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:13.352 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-03-25T18:38:13.549 DEBUG:teuthology.parallel:result is None 2024-03-25T18:38:13.550 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:38:13.693 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:38:13.713 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:38:13.713 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:38:13.713 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:38:13.715 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:38:13.860 DEBUG:teuthology.orchestra.run.smithi136:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:38:13.884 INFO:teuthology.orchestra.run.smithi136.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:38:13.885 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:38:13.885 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:38:13.887 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:38:13.887 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:13.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:38:13.914 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:13.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:38:13.955 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:38:13.955 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:13.955 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:38:13.983 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:38:14.049 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:14.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:38:14.079 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:38:14.148 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:38:14.148 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:14.148 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:38:14.176 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:38:14.243 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:14.243 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:38:14.271 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:38:14.339 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:38:14.339 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:14.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:38:14.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:38:14.432 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:14.432 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:38:14.460 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:38:14.528 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:38:14.612 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:38:14.612 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /var/log/ceph 2024-03-25T18:38:14.614 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 777 /var/log/ceph 2024-03-25T18:38:14.642 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:38:14.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:38:14.681 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:38:14.709 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:38:14.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:38:14.748 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:38:14.813 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:38:14.813 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:38:14.813 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621284', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi032.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQ2cEhYIhasdCSbjjrbMgx4YxiqTKlERSjnvWz3CHmGkjcWVq30z9d3cBowRGiDQ5TgJ7u9R8Ifp+cYmJCQMH0=', 'smithi136.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPAOGTiEUOU1pEwviVaRkC2dui5+QB0HeHVyRSXf0gqxq6Z2loYjA6tAp6VkzPXtjLroU92ffcKbBWZ8ZttYcw='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:38:14.814 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:38:14.816 DEBUG:teuthology.orchestra.run.smithi136:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:38:14.831 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:38:14.870 DEBUG:teuthology.orchestra.run.smithi136:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:38:14.898 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:14.898 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:38:14.950 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:38:14.950 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-25 17:16:04.466352995 +0000 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-25 17:08:14.197274700 +0000 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-25 17:08:14.197274700 +0000 2024-03-25T18:38:15.004 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-25 17:08:14.197274700 +0000 2024-03-25T18:38:15.005 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:38:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-25T18:38:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-25T18:38:15.069 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 8.2073e-05 s, 6.2 MB/s 2024-03-25T18:38:15.070 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:38:15.125 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-03-25T18:38:15.179 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:38:15.179 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:38:15.179 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:38:15.179 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:38:15.179 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:38:15.179 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-25 17:16:04.467352974 +0000 2024-03-25T18:38:15.179 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-25 17:08:14.549267195 +0000 2024-03-25T18:38:15.180 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-25 17:08:14.549267195 +0000 2024-03-25T18:38:15.180 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-25 17:08:14.549267195 +0000 2024-03-25T18:38:15.180 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:38:15.243 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-25T18:38:15.243 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-25T18:38:15.244 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.4255e-05 s, 5.4 MB/s 2024-03-25T18:38:15.245 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:38:15.300 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-03-25T18:38:15.354 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:38:15.354 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:38:15.354 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-25T18:38:15.354 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:38:15.354 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:38:15.354 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-25 17:16:04.467352974 +0000 2024-03-25T18:38:15.354 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-25 17:08:14.897259775 +0000 2024-03-25T18:38:15.355 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-25 17:08:14.897259775 +0000 2024-03-25T18:38:15.355 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-25 17:08:14.897259775 +0000 2024-03-25T18:38:15.355 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:38:15.418 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-25T18:38:15.419 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-25T18:38:15.419 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 8.8736e-05 s, 5.8 MB/s 2024-03-25T18:38:15.419 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:38:15.477 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-03-25 17:16:04.468352953 +0000 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-03-25 17:08:15.228252717 +0000 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-03-25 17:08:15.228252717 +0000 2024-03-25T18:38:15.529 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-03-25 17:08:15.228252717 +0000 2024-03-25T18:38:15.530 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:38:15.593 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-03-25T18:38:15.593 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-03-25T18:38:15.593 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 9.3977e-05 s, 5.4 MB/s 2024-03-25T18:38:15.594 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:38:15.649 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:15.650 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:15.650 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:38:15.667 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:38:15.667 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_1 2024-03-25T18:38:15.724 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:38:15.724 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:38:15.724 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:38:15.724 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:38:15.724 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:38:15.724 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-25 17:15:33.319656159 +0000 2024-03-25T18:38:15.724 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-25 17:08:07.715047515 +0000 2024-03-25T18:38:15.724 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-25 17:08:07.715047515 +0000 2024-03-25T18:38:15.725 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-25 17:08:07.715047515 +0000 2024-03-25T18:38:15.725 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:38:15.792 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-25T18:38:15.792 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-25T18:38:15.792 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000232656 s, 2.2 MB/s 2024-03-25T18:38:15.794 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:38:15.852 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_2 2024-03-25T18:38:15.909 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:38:15.909 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:38:15.909 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-25T18:38:15.909 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:38:15.909 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:38:15.909 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-25 17:15:33.319656159 +0000 2024-03-25T18:38:15.909 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-25 17:08:08.058039499 +0000 2024-03-25T18:38:15.909 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-25 17:08:08.058039499 +0000 2024-03-25T18:38:15.910 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-25 17:08:08.058039499 +0000 2024-03-25T18:38:15.910 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:38:15.977 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-25T18:38:15.977 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-25T18:38:15.977 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000248332 s, 2.1 MB/s 2024-03-25T18:38:15.979 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:38:16.036 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_3 2024-03-25T18:38:16.092 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:38:16.092 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:38:16.092 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:38:16.092 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:38:16.092 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:38:16.093 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-25 17:15:33.319656159 +0000 2024-03-25T18:38:16.093 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-25 17:08:08.406031367 +0000 2024-03-25T18:38:16.093 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-25 17:08:08.406031367 +0000 2024-03-25T18:38:16.093 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-25 17:08:08.406031367 +0000 2024-03-25T18:38:16.093 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:38:16.159 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-25T18:38:16.159 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-25T18:38:16.159 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000217533 s, 2.4 MB/s 2024-03-25T18:38:16.160 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:38:16.216 DEBUG:teuthology.orchestra.run.smithi136:> stat /dev/vg_nvme/lv_4 2024-03-25T18:38:16.271 INFO:teuthology.orchestra.run.smithi136.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:38:16.272 INFO:teuthology.orchestra.run.smithi136.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:38:16.272 INFO:teuthology.orchestra.run.smithi136.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-25T18:38:16.272 INFO:teuthology.orchestra.run.smithi136.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:38:16.272 INFO:teuthology.orchestra.run.smithi136.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:38:16.272 INFO:teuthology.orchestra.run.smithi136.stdout:Access: 2024-03-25 17:15:33.320656137 +0000 2024-03-25T18:38:16.272 INFO:teuthology.orchestra.run.smithi136.stdout:Modify: 2024-03-25 17:08:08.756023188 +0000 2024-03-25T18:38:16.272 INFO:teuthology.orchestra.run.smithi136.stdout:Change: 2024-03-25 17:08:08.756023188 +0000 2024-03-25T18:38:16.272 INFO:teuthology.orchestra.run.smithi136.stdout: Birth: 2024-03-25 17:08:08.756023188 +0000 2024-03-25T18:38:16.272 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:38:16.336 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records in 2024-03-25T18:38:16.337 INFO:teuthology.orchestra.run.smithi136.stderr:1+0 records out 2024-03-25T18:38:16.337 INFO:teuthology.orchestra.run.smithi136.stderr:512 bytes copied, 0.000219636 s, 2.3 MB/s 2024-03-25T18:38:16.338 DEBUG:teuthology.orchestra.run.smithi136:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:38:16.393 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:16.394 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:38:16.394 INFO:tasks.ceph:Generating config... 2024-03-25T18:38:16.395 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:38:16.395 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:38:16.395 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:38:16.395 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:38:16.395 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:38:16.395 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:38:16.395 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:38:16.395 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:38:16.395 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:38:16.395 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:38:16.396 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:38:16.396 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:38:16.396 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:38:16.396 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:38:16.396 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:38:16.397 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:38:16.398 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:38:16.398 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:38:16.398 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:38:16.451 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:38:16.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:38:16.502 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:38:16.529 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.32:6789]'), ('mon.c', '[v1:172.21.15.32:6790]'), ('mon.b', '[v1:172.21.15.136:6789]')] 2024-03-25T18:38:16.530 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.32:6789],[v1:172.21.15.32:6790],[v1:172.21.15.136:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:38:16.531 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:16.531 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:38:16.585 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.32:6789]' --addv c '[v1:172.21.15.32:6790]' --addv b '[v1:172.21.15.136:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:16.671 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:16.671 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: generated fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:38:16.671 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-03-25T18:38:16.671 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 0 2024-03-25T18:38:16.671 INFO:teuthology.orchestra.run.smithi032.stdout:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:38:16.671 INFO:teuthology.orchestra.run.smithi032.stdout:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:38:16.672 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:38:16.672 INFO:teuthology.orchestra.run.smithi032.stdout:min_mon_release 17 (quincy) 2024-03-25T18:38:16.672 INFO:teuthology.orchestra.run.smithi032.stdout:election_strategy: 3 2024-03-25T18:38:16.672 INFO:teuthology.orchestra.run.smithi032.stdout:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:38:16.672 INFO:teuthology.orchestra.run.smithi032.stdout:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:38:16.672 INFO:teuthology.orchestra.run.smithi032.stdout:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:38:16.672 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:38:16.673 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:38:16.687 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID d7917469-fe02-4321-912b-a3ddb3c8ed96... 2024-03-25T18:38:16.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:38:16.730 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:38:16.777 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: chdir = "" 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: auth supported = cephx 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: filestore xattr use omap = true 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: mon clock drift allowed = 1.000 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: osd crush chooseleaf type = 0 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: auth debug = true 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on old message = true 2024-03-25T18:38:16.778 INFO:teuthology.orchestra.run.smithi032.stdout: ms die on bug = true 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon pg warn max object skew = 0 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default size = 2 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow primary affinity = true 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd allow pg remap = true 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on no sortbitwise = false 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:38:16.779 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on too few osds = false 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon_allow_pool_size_one = true 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: osd default data pool replay window = 5 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon allow pool delete = true 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon cluster log file level = debug 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: debug asserts on shutdown = true 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon health detail to clog = false 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon host = "[v1:172.21.15.32:6789],[v1:172.21.15.32:6790],[v1:172.21.15.136:6789]" 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon client directed command retry = 5 2024-03-25T18:38:16.780 INFO:teuthology.orchestra.run.smithi032.stdout: mon election default strategy = 3 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: ms bind msgr2 = False 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: ms inject socket failures = 5000 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: ms type = async 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: osd heartbeat grace = 80 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout:[osd] 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: osd journal size = 100 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub load threshold = 5.0 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: osd scrub max interval = 600 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: osd recover clone overlap = true 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: osd recovery max chunk = 1048576 2024-03-25T18:38:16.781 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug shutdown = true 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug op order = true 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify stray on activate = true 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug trim objects = true 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: osd open classes on start = true 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug pg log writeout = true 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: osd map max advance = 10 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: journal zero on create = true 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.782 INFO:teuthology.orchestra.run.smithi032.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: filestore apply finisher threads = 3 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: bdev debug aio = true 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug misdirected ops = true 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: bdev async discard = True 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: bdev enable discard = True 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore allocator = stupid 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore block size = 96636764160 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore fsck on mount = True 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: bluestore zero block detection = True 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluefs = 1 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: debug bluestore = 1 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: debug monc = 20 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-25T18:38:16.783 INFO:teuthology.orchestra.run.smithi032.stdout: debug osd = 20 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: debug rocksdb = 10 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd full ratio = 0.9 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd class default list = * 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd class load list = * 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd client watch timeout = 120 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify cached snaps = True 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd debug verify missing on start = True 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd fast shutdown = False 2024-03-25T18:38:16.784 INFO:teuthology.orchestra.run.smithi032.stdout: osd max markdown count = 1000 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: osd mclock override recovery settings = True 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: osd objectstore = bluestore 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: osd op queue = debug_random 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: osd op queue cut off = debug_random 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: osd shutdown pgref assert = True 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr] 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: debug mgr = 20 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:38:16.785 INFO:teuthology.orchestra.run.smithi032.stdout: mgr/telemetry/nag = false 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout:[mon] 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: debug mon = 20 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: debug paxos = 20 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: debug auth = 20 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: mon data avail warn = 5 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: mon mgr mkfs grace = 240 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd reporter subtree level = osd 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd prime pg temp = true 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.786 INFO:teuthology.orchestra.run.smithi032.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: # 1m isn't quite enough 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon min osdmap epochs = 50 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon osd crush smoke test = False 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon osdmap full prune interval = 2 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon osdmap full prune min = 15 2024-03-25T18:38:16.787 INFO:teuthology.orchestra.run.smithi032.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: mon scrub interval = 300 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: mon warn on pool no app = False 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: paxos service trim min = 10 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout:[client] 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: rgw cache enabled = true 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable ops log = true 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: rgw enable usage log = true 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: debug monc = 20 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: debug ms = 1 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: debug objecter = 20 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout: debug rados = 20 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.a] 2024-03-25T18:38:16.788 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.c] 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi032.stdout:[mon.b] 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout:[global] 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout: chdir = "" 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout: auth supported = cephx 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout: filestore xattr use omap = true 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout: mon clock drift allowed = 1.000 2024-03-25T18:38:16.789 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: osd crush chooseleaf type = 0 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: auth debug = true 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on old message = true 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: ms die on bug = true 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: mon pg warn max object skew = 0 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default size = 2 2024-03-25T18:38:16.790 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow primary affinity = true 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd allow pg remap = true 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on no sortbitwise = false 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on too few osds = false 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: mon_allow_pool_size_one = true 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: osd default data pool replay window = 5 2024-03-25T18:38:16.791 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: mon allow pool delete = true 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: mon cluster log file level = debug 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: debug asserts on shutdown = true 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: mon health detail to clog = false 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: mon host = "[v1:172.21.15.32:6789],[v1:172.21.15.32:6790],[v1:172.21.15.136:6789]" 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: mon client directed command retry = 5 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: mon election default strategy = 3 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: ms bind msgr2 = False 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: ms inject socket failures = 5000 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: ms type = async 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: osd heartbeat grace = 80 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: fsid = d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:38:16.792 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout:[osd] 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd journal size = 100 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub load threshold = 5.0 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd scrub max interval = 600 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd recover clone overlap = true 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd recovery max chunk = 1048576 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug shutdown = true 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug op order = true 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify stray on activate = true 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug trim objects = true 2024-03-25T18:38:16.793 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: osd open classes on start = true 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug pg log writeout = true 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: osd map max advance = 10 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: journal zero on create = true 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: filestore apply finisher threads = 3 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: bdev debug aio = true 2024-03-25T18:38:16.794 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug misdirected ops = true 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: bdev async discard = True 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: bdev enable discard = True 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore allocator = stupid 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore block size = 96636764160 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore fsck on mount = True 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: bluestore zero block detection = True 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluefs = 1 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: debug bluestore = 1 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: debug monc = 20 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: debug osd = 20 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: debug rocksdb = 10 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:38:16.795 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd full ratio = 0.9 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd class default list = * 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd class load list = * 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd client watch timeout = 120 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify cached snaps = True 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd debug verify missing on start = True 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd fast shutdown = False 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd max markdown count = 1000 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd mclock override recovery settings = True 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd objectstore = bluestore 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue = debug_random 2024-03-25T18:38:16.796 INFO:teuthology.orchestra.run.smithi136.stdout: osd op queue cut off = debug_random 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: osd shutdown pgref assert = True 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr] 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: debug mgr = 20 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: mgr/telemetry/nag = false 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout:[mon] 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: debug mon = 20 2024-03-25T18:38:16.797 INFO:teuthology.orchestra.run.smithi136.stdout: debug paxos = 20 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: debug auth = 20 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: mon data avail warn = 5 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: mon mgr mkfs grace = 240 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd reporter subtree level = osd 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd prime pg temp = true 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:38:16.798 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: # 1m isn't quite enough 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon min osdmap epochs = 50 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon osd crush smoke test = False 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune interval = 2 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune min = 15 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon scrub interval = 300 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: mon warn on pool no app = False 2024-03-25T18:38:16.799 INFO:teuthology.orchestra.run.smithi136.stdout: paxos service trim min = 10 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout:[client] 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: rgw cache enabled = true 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable ops log = true 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: rgw enable usage log = true 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: debug monc = 20 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: debug ms = 1 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: debug objecter = 20 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout: debug rados = 20 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.a] 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.c] 2024-03-25T18:38:16.800 INFO:teuthology.orchestra.run.smithi136.stdout:[mon.b] 2024-03-25T18:38:16.802 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:38:16.802 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:38:16.857 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:38:16.857 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:16.857 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:38:16.871 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:16.871 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:38:16.926 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:38:16.927 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:16.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:38:16.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:38:17.005 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:17.005 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:17.060 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:38:17.060 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:17.060 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:38:17.061 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:38:17.102 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:17.102 DEBUG:teuthology.orchestra.run.smithi136:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:17.157 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:38:17.157 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:38:17.158 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:38:17.211 INFO:teuthology.orchestra.run.smithi032.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:38:17.214 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:38:17.276 INFO:teuthology.orchestra.run.smithi136.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:38:17.278 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:38:17.278 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:38:17.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:38:17.320 INFO:teuthology.orchestra.run.smithi032.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:38:17.335 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:38:17.377 INFO:teuthology.orchestra.run.smithi136.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:38:17.392 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:38:17.393 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi032.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi136.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:38:17.393 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:38:17.419 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:17.419 INFO:tasks.ceph:role: osd.0 2024-03-25T18:38:17.419 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:38:17.419 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:38:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:38:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:38:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:38:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:38:17.484 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:38:17.484 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:38:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:38:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:38:17.485 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:38:17.485 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:38:17.523 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-25T18:38:17.525 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-25T18:38:17.526 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:38:17.597 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:38:17.669 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:38:17.749 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:17.750 INFO:tasks.ceph:role: osd.1 2024-03-25T18:38:17.750 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:38:17.750 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:38:17.818 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:38:17.849 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-25T18:38:17.851 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-25T18:38:17.852 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:38:17.923 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:38:17.996 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:38:18.063 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:18.063 INFO:tasks.ceph:role: osd.2 2024-03-25T18:38:18.063 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:38:18.063 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:38:18.130 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:38:18.164 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-25T18:38:18.166 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-25T18:38:18.166 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:38:18.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:38:18.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:38:18.376 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:18.376 INFO:tasks.ceph:role: osd.3 2024-03-25T18:38:18.376 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:38:18.376 DEBUG:teuthology.orchestra.run.smithi032:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:38:18.442 INFO:teuthology.orchestra.run.smithi032.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:38:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:38:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:38:18.442 INFO:teuthology.orchestra.run.smithi032.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:38:18.442 INFO:teuthology.orchestra.run.smithi032.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:38:18.443 INFO:teuthology.orchestra.run.smithi032.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:38:18.443 INFO:teuthology.orchestra.run.smithi032.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:38:18.443 INFO:teuthology.orchestra.run.smithi032.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:38:18.443 INFO:teuthology.orchestra.run.smithi032.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:38:18.443 INFO:teuthology.orchestra.run.smithi032.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:38:18.477 INFO:teuthology.orchestra.run.smithi032.stdout:Discarding blocks...Done. 2024-03-25T18:38:18.479 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi032.front.sepia.ceph.com -o noatime 2024-03-25T18:38:18.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:38:18.555 DEBUG:teuthology.orchestra.run.smithi032:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:38:18.627 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:18.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:18.723+0000 7fa2fe219640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:38:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:18.723+0000 7fa2fe219640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:38:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:18.723+0000 7fa2fe219640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:38:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:18.723+0000 7fa2fe219640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:38:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:18.723+0000 7fa2fe219640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:38:18.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:18.723+0000 7fa2fe219640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:38:21.567 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:38:21.598 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:21.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:21.688+0000 7ff245a0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:38:21.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:21.689+0000 7ff245a0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:38:21.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:21.689+0000 7ff245a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:38:21.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:21.689+0000 7ff245a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:38:21.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:21.689+0000 7ff245a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:38:21.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:21.689+0000 7ff245a0d640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:38:24.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:38:24.560 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:24.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:24.650+0000 7f46f2001640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:38:24.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:24.650+0000 7f46f2001640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:38:24.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:24.650+0000 7f46f2001640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:38:24.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:24.650+0000 7f46f2001640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:38:24.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:24.650+0000 7f46f2001640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:38:24.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:24.651+0000 7f46f2001640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:38:27.498 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:38:27.529 DEBUG:teuthology.orchestra.run.smithi032:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:27.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:27.620+0000 7fbe55c12640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:38:27.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:27.620+0000 7fbe55c12640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:38:27.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:27.620+0000 7fbe55c12640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:38:27.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:27.620+0000 7fbe55c12640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:38:27.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:27.620+0000 7fbe55c12640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:38:27.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:27.620+0000 7fbe55c12640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:38:30.461 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:38:30.492 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:38:30.523 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:30.523 INFO:tasks.ceph:role: osd.4 2024-03-25T18:38:30.524 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:38:30.524 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:38:30.601 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:38:30.602 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:38:31.112 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-25T18:38:31.114 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-25T18:38:31.114 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:38:31.208 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:38:31.244 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:38:31.313 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:31.313 INFO:tasks.ceph:role: osd.5 2024-03-25T18:38:31.313 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:38:31.313 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:38:31.389 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:38:31.389 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:38:31.389 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:38:31.389 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:38:31.389 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:38:31.389 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:38:31.389 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:38:31.390 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:38:31.390 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:38:31.390 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:38:31.902 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-25T18:38:31.905 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-25T18:38:31.905 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:38:32.015 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:38:32.050 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:38:32.119 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:32.119 INFO:tasks.ceph:role: osd.6 2024-03-25T18:38:32.119 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:38:32.119 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:38:32.195 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:38:32.195 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:38:32.195 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:38:32.195 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:38:32.195 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:38:32.195 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:38:32.195 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:38:32.195 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:38:32.196 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:38:32.196 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:38:32.730 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-25T18:38:32.733 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-25T18:38:32.733 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:38:32.843 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:38:32.879 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:38:32.948 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:38:32.949 INFO:tasks.ceph:role: osd.7 2024-03-25T18:38:32.949 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:38:32.949 DEBUG:teuthology.orchestra.run.smithi136:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:38:33.025 INFO:teuthology.orchestra.run.smithi136.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:38:33.025 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:38:33.025 INFO:teuthology.orchestra.run.smithi136.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:38:33.025 INFO:teuthology.orchestra.run.smithi136.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:38:33.025 INFO:teuthology.orchestra.run.smithi136.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:38:33.025 INFO:teuthology.orchestra.run.smithi136.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:38:33.025 INFO:teuthology.orchestra.run.smithi136.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:38:33.025 INFO:teuthology.orchestra.run.smithi136.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:38:33.026 INFO:teuthology.orchestra.run.smithi136.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:38:33.026 INFO:teuthology.orchestra.run.smithi136.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:38:33.543 INFO:teuthology.orchestra.run.smithi136.stdout:Discarding blocks...Done. 2024-03-25T18:38:33.545 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi136.front.sepia.ceph.com -o noatime 2024-03-25T18:38:33.545 DEBUG:teuthology.orchestra.run.smithi136:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:38:33.640 DEBUG:teuthology.orchestra.run.smithi136:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:38:33.676 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:33.777 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:33.777+0000 7f1fcd6b5640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:38:33.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:33.778+0000 7f1fcd6b5640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:38:33.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:33.778+0000 7f1fcd6b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:38:33.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:33.778+0000 7f1fcd6b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:38:33.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:33.778+0000 7f1fcd6b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:38:33.779 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:33.779+0000 7f1fcd6b5640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:38:36.631 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:38:36.662 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:36.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:36.752+0000 7fbb1f5ff640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:38:36.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:36.752+0000 7fbb1f5ff640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:38:36.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:36.752+0000 7fbb1f5ff640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:38:36.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:36.752+0000 7fbb1f5ff640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:38:36.753 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:36.752+0000 7fbb1f5ff640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:38:36.754 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:36.753+0000 7fbb1f5ff640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:38:39.595 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:38:39.627 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:39.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:39.717+0000 7f7223bf8640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:38:39.718 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:39.718+0000 7f7223bf8640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:38:39.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:39.718+0000 7f7223bf8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:38:39.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:39.718+0000 7f7223bf8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:38:39.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:39.718+0000 7f7223bf8640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:38:39.719 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:39.718+0000 7f7223bf8640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:38:42.561 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:38:42.593 DEBUG:teuthology.orchestra.run.smithi136:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:42.684 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:42.683+0000 7fb670415640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:38:42.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:42.684+0000 7fb670415640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:38:42.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:42.684+0000 7fb670415640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:38:42.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:42.684+0000 7fb670415640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:38:42.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:42.684+0000 7fb670415640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:38:42.685 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:38:42.684+0000 7fb670415640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:38:45.526 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:38:45.558 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:38:45.558 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:45.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:38:45.590 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:45.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:38:45.663 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:45.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:38:45.734 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:45.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:38:45.803 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:45.803 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:38:45.872 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:45.872 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:38:45.904 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:45.904 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:38:45.974 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:45.975 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:38:46.040 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:46.041 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:38:46.107 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:46.107 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:38:46.177 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:38:46.177 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:38:46.195 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:38:46.195 DEBUG:teuthology.orchestra.run.smithi136:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:38:46.229 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:38:46.229 DEBUG:teuthology.orchestra.run.smithi032:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:38:46.237 DEBUG:teuthology.orchestra.run.smithi136:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:38:46.272 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.y] 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCZxAFmwnuoDBAAjAXBe88buX6enbo+CBdvKA== 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.0] 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCaxAFmfQMzKxAAd0iiM31/btKC8+r9WXhkxg== 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.1] 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCdxAFmmlglKRAAtxjcH7Sy7fuJVPeBx4mKvw== 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.2] 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCgxAFmwsffJhAAQXoFSg79nMAjzS6Q7/sjiQ== 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.3] 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCjxAFmBD79JBAA9kBwJkxUbiKIrfboQ41img== 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout:[mgr.x] 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCZxAFmwyx3EBAA6IV8HqOKW+6e8MtxDQqOag== 2024-03-25T18:38:46.273 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.4] 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCpxAFmNRNfLhAAJGAJKYsTTsnuA21+QVE5eg== 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.5] 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCsxAFmhJPjLBAAukP2gh1gQayNzkMrtLv1TA== 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.6] 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCvxAFm7UbVKhAARh4XrtJuPLiTpkNa/Qkv9Q== 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout:[osd.7] 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCyxAFmOn/PKBAA2o8EQwqZ444HZn5X3ln3YQ== 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCZxAFmvYofExAA+pCnp4j/WDbdd28aeHnGtg== 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout:[client.1] 2024-03-25T18:38:46.274 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCZxAFm+u+IFhAA3CuVOUTh11fz6/ASgoWSsg== 2024-03-25T18:38:46.295 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.y] 2024-03-25T18:38:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCZxAFmwnuoDBAAjAXBe88buX6enbo+CBdvKA== 2024-03-25T18:38:46.295 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.0] 2024-03-25T18:38:46.295 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCaxAFmfQMzKxAAd0iiM31/btKC8+r9WXhkxg== 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.1] 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCdxAFmmlglKRAAtxjcH7Sy7fuJVPeBx4mKvw== 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.2] 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCgxAFmwsffJhAAQXoFSg79nMAjzS6Q7/sjiQ== 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.3] 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCjxAFmBD79JBAA9kBwJkxUbiKIrfboQ41img== 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout:[mgr.x] 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCZxAFmwyx3EBAA6IV8HqOKW+6e8MtxDQqOag== 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.4] 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCpxAFmNRNfLhAAJGAJKYsTTsnuA21+QVE5eg== 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.5] 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCsxAFmhJPjLBAAukP2gh1gQayNzkMrtLv1TA== 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.6] 2024-03-25T18:38:46.296 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCvxAFm7UbVKhAARh4XrtJuPLiTpkNa/Qkv9Q== 2024-03-25T18:38:46.297 INFO:teuthology.orchestra.run.smithi136.stdout:[osd.7] 2024-03-25T18:38:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCyxAFmOn/PKBAA2o8EQwqZ444HZn5X3ln3YQ== 2024-03-25T18:38:46.297 INFO:teuthology.orchestra.run.smithi136.stdout:[client.0] 2024-03-25T18:38:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCZxAFmvYofExAA+pCnp4j/WDbdd28aeHnGtg== 2024-03-25T18:38:46.297 INFO:teuthology.orchestra.run.smithi136.stdout:[client.1] 2024-03-25T18:38:46.297 INFO:teuthology.orchestra.run.smithi136.stdout: key = AQCZxAFm+u+IFhAA3CuVOUTh11fz6/ASgoWSsg== 2024-03-25T18:38:46.298 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:38:46.319 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:38:46.392 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.394 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.447 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.449 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.500 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.503 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.556 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.558 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.611 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:38:46.613 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:38:46.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.667 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.718 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.721 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.773 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.776 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.829 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:38:46.880 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:38:46.883 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:38:46.937 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:38:46.939 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:38:46.994 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:38:46.995 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:38:47.021 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:38:47.117 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:38:47.144 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:38:47.210 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:38:47.306 DEBUG:teuthology.orchestra.run.smithi032:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:38:47.332 DEBUG:teuthology.orchestra.run.smithi136:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:38:47.360 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:38:47.457 DEBUG:teuthology.orchestra.run.smithi136:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:38:47.484 DEBUG:teuthology.orchestra.run.smithi032:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:47.486 DEBUG:teuthology.orchestra.run.smithi136:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:38:47.539 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:38:47.539 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:38:47.539 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:38:47.539 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:38:47.542 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:38:47.542 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:38:47.542 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:38:47.542 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:38:47.544 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:38:47.544 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:38:47.544 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:38:47.545 DEBUG:teuthology.orchestra.run.smithi136:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:38:47.582 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:38:47.582 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:38:47.582 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:38:47.582 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:38:47.585 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:38:47.585 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:38:47.585 DEBUG:teuthology.orchestra.run.smithi136:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:38:47.587 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:38:47.587 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:38:47.587 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:38:47.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.687+0000 7fe9a6182640 10 monclient: get_monmap_and_config 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.687+0000 7fe9a6182640 10 monclient: build_initial_monmap 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.687+0000 7fe9a6182640 10 monclient: monmap: 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:47.688588+0000 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:47.688588+0000 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.687+0000 7fe9a6182640 1 Processor -- start 2024-03-25T18:38:47.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.687+0000 7fe9a6182640 1 -- start start 2024-03-25T18:38:47.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.687+0000 7fe9a6182640 10 monclient: init 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 10 monclient: _reopen_session rank -1 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 10 monclient(hunting): picked mon.noname-a con 0x7fe9a00b4700 addr v1:172.21.15.32:6789/0 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a0142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a00a96b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00a9d60 con 0x7fe9a00b4700 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a013ff10 con 0x7fe9a0142a20 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00aa1f0 con 0x7fe9a00a96b0 2024-03-25T18:38:47.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:47.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe9a6182640 10 monclient(hunting): authenticate will time out at 6033.878077s 2024-03-25T18:38:47.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe99ffff640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe9a013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:47.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe9a00b4700 legacy=0x7fe9a0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:47.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.688+0000 7fe99effd640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe9a013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:47.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.889+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe9a00b4700 legacy=0x7fe9a0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:47.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.889+0000 7fe99ffff640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe9a013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:47.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:47.889+0000 7fe99effd640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe9a013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:48.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:48.290+0000 7fe99ffff640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe9a013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:48.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:48.290+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe9a00b4700 legacy=0x7fe9a0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:48.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:48.290+0000 7fe99effd640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe9a013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:49.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:49.091+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe9a00b4700 legacy=0x7fe9a0142310 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:49.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:49.091+0000 7fe99ffff640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe9a013f800 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:49.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:49.091+0000 7fe99effd640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe9a013f0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 10 monclient(hunting): tick 2024-03-25T18:38:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:38:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:38:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:38:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe9a013f800 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe9a013f0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:50.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe9a00b4700 legacy=0x7fe9a0142310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c003680 addr v1:172.21.15.32:6789/0 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a0142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a00a96b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00b4ae0 con 0x7fe98c003680 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00a9d60 con 0x7fe9a0142a20 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a013ff10 con 0x7fe9a00a96b0 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99ffff640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:50.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.688+0000 7fe99effd640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:50.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.889+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:50.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.889+0000 7fe99ffff640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:50.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:50.889+0000 7fe99effd640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:51.290+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:51.290+0000 7fe99ffff640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:51.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:51.290+0000 7fe99effd640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:52.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:52.091+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c005ad0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:52.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:52.091+0000 7fe99ffff640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:52.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:52.091+0000 7fe99effd640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:53.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 10 monclient(hunting): tick 2024-03-25T18:38:53.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:38:53.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c005ad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c003680 addr v1:172.21.15.32:6789/0 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a00a96b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a0142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00aa1f0 con 0x7fe98c003680 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00b4ae0 con 0x7fe9a0142a20 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00a9d60 con 0x7fe9a00a96b0 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99effd640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:53.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.688+0000 7fe99ffff640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:53.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.889+0000 7fe99effd640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:53.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.889+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:53.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:53.890+0000 7fe99ffff640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:54.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:54.290+0000 7fe99effd640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:54.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:54.290+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:54.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:54.291+0000 7fe99ffff640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:54.575 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:00:06.892 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:54.884 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:00:07.201 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:55.072 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:00:07.400 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:55.076 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:00:07.404 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:55.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:55.091+0000 7fe99effd640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6790/0 2024-03-25T18:38:55.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:55.091+0000 7fe99f7fe640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c010430 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.32:6789/0 2024-03-25T18:38:55.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:55.092+0000 7fe99ffff640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.136:6789/0 2024-03-25T18:38:55.400 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:00:07.728 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:55.403 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:00:07.731 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:38:56.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.688+0000 7fe99d7fa640 10 monclient(hunting): tick 2024-03-25T18:38:56.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:38:56.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:38:56.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 1 -- >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c00cbf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 1 -- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c009300 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 1 -- >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c010430 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c003680 addr v1:172.21.15.32:6789/0 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a00a96b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fe9a0142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a013ff10 con 0x7fe98c003680 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00aa1f0 con 0x7fe9a0142a20 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00b4ae0 con 0x7fe9a00a96b0 2024-03-25T18:38:56.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.689+0000 7fe99d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:38:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.746+0000 7fe99ffff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 0x7fe98c010c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:40904/0 (socket says 172.21.15.32:40904) 2024-03-25T18:38:56.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:56.746+0000 7fe99ffff640 1 -- 172.21.15.32:0/3984609240 learned_addr learned my addr 172.21.15.32:0/3984609240 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:38:57.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:57.067+0000 7fe99effd640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c006af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-25T18:38:57.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:57.067+0000 7fe99effd640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c006af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:38:57.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:57.067+0000 7fe99effd640 1 --1- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 0x7fe98c006af0 :-1 s=OPENED pgs=1 cs=1 l=1).handle_message read tag failed 2024-03-25T18:38:57.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:57.067+0000 7fe99effd640 1 --1- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 0x7fe98c006af0 :-1 s=OPENED pgs=1 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:38:57.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:57.067+0000 7fe99e7fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.32:6790/0 2024-03-25T18:38:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:57.116+0000 7fe99f7fe640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:38:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:57.116+0000 7fe99f7fe640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:38:57.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:57.116+0000 7fe99f7fe640 1 --1- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 0x7fe98c010430 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:38:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 10 monclient(hunting): tick 2024-03-25T18:38:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:38:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:38:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:38:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c006af0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:38:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.136:6789/0 conn(0x7fe9a0142a20 legacy=0x7fe98c010c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6789/0 conn(0x7fe98c003680 legacy=0x7fe98c010430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:38:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:38:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fe98c009320 addr v1:172.21.15.136:6789/0 2024-03-25T18:38:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe9a00a96b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:38:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c010c00 addr v1:172.21.15.32:6789/0 2024-03-25T18:38:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00a9d60 con 0x7fe98c010c00 2024-03-25T18:38:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c010fe0 con 0x7fe98c009320 2024-03-25T18:38:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c011170 con 0x7fe9a00a96b0 2024-03-25T18:38:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:38:59.689+0000 7fe99d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 10 monclient(hunting): tick 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe9a00a96b0 legacy=0x7fe98c006af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.136:6789/0 conn(0x7fe98c009320 legacy=0x7fe98c00cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6789/0 conn(0x7fe98c010c00 legacy=0x7fe98c003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe98c0106b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:02.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fe98c013510 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c010c00 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c0104d0 con 0x7fe98c010c00 2024-03-25T18:39:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c006b40 con 0x7fe98c013510 2024-03-25T18:39:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c006d70 con 0x7fe98c0106b0 2024-03-25T18:39:02.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:02.689+0000 7fe99d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 10 monclient(hunting): tick 2024-03-25T18:39:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:39:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:39:05.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe98c0106b0 legacy=0x7fe98c004170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.136:6789/0 conn(0x7fe98c013510 legacy=0x7fe98c015900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6789/0 conn(0x7fe98c010c00 legacy=0x7fe98c003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe98c0114f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c012060 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fe98c013510 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c006fa0 con 0x7fe98c012060 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c004170 con 0x7fe98c013510 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c0043a0 con 0x7fe98c0114f0 2024-03-25T18:39:05.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:05.689+0000 7fe99d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:08.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 10 monclient(hunting): tick 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 1 monclient(hunting): continuing hunt 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe98c0114f0 legacy=0x7fe98c011950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.136:6789/0 conn(0x7fe98c013510 legacy=0x7fe98c003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6789/0 conn(0x7fe98c012060 legacy=0x7fe98c00cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-b con 0x7fe98c009a30 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c00a540 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 10 monclient(hunting): picked mon.noname-c con 0x7fe98c013510 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c0045d0 con 0x7fe98c00a540 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c0118d0 con 0x7fe98c013510 2024-03-25T18:39:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe98c011ab0 con 0x7fe98c009a30 2024-03-25T18:39:08.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:08.690+0000 7fe99d7fa640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:11.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.069+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336043511 0 0) 0x7fe98c0045d0 con 0x7fe98c00a540 2024-03-25T18:39:11.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.069+0000 7fe99e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:11.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.069+0000 7fe99e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:11.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.069+0000 7fe99e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:11.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.069+0000 7fe99e7fc640 10 monclient(hunting): my global_id is 4101 2024-03-25T18:39:11.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.069+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe988003520 con 0x7fe98c00a540 2024-03-25T18:39:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.104+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1201986663 0 0) 0x7fe98c011ab0 con 0x7fe98c009a30 2024-03-25T18:39:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.104+0000 7fe99e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.104+0000 7fe99e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.104+0000 7fe99e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:11.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.104+0000 7fe99e7fc640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:39:11.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.104+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c0045d0 con 0x7fe98c009a30 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3520198159 0 0) 0x7fe988003520 con 0x7fe98c00a540 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6790/0 conn(0x7fe98c009a30 legacy=0x7fe98c009e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.136:6789/0 conn(0x7fe98c013510 legacy=0x7fe98c003a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98c011ce0 con 0x7fe98c00a540 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:39:11.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe994003ef0 con 0x7fe98c00a540 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe99e7fc640 10 monclient: dump: 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe9a6182640 5 monclient: authenticate success, global_id 4101 2024-03-25T18:39:11.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.124+0000 7fe9a6182640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe994004050 con 0x7fe98c00a540 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe99e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe99e7fc640 1 -- 172.21.15.32:0/3984609240 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe994004460 con 0x7fe98c00a540 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 4 set_mon_vals no callback set 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe99e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe99e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe99e7fc640 10 monclient: dump: 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:11.153 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:11.154 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:11.154 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:11.154 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:11.154 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:11.154 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:11.154 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 10 monclient: shutdown 2024-03-25T18:39:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 1 -- 172.21.15.32:0/3984609240 >> v1:172.21.15.32:6789/0 conn(0x7fe98c00a540 legacy=0x7fe98c015900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 1 -- 172.21.15.32:0/3984609240 shutdown_connections 2024-03-25T18:39:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 1 -- 172.21.15.32:0/3984609240 >> 172.21.15.32:0/3984609240 conn(0x7fe9a00668e0 msgr2=0x7fe9a0066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 1 -- 172.21.15.32:0/3984609240 shutdown_connections 2024-03-25T18:39:11.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.152+0000 7fe9a6182640 1 -- 172.21.15.32:0/3984609240 wait complete. 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.153+0000 7fe9a6182640 10 monclient: build_initial_monmap 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.153+0000 7fe9a6182640 10 monclient: monmap: 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.153+0000 7fe9a6182640 1 librados: starting msgr at 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.153+0000 7fe9a6182640 1 librados: starting objecter 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.153+0000 7fe9a6182640 1 Processor -- start 2024-03-25T18:39:11.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 1 -- start start 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 1 librados: setting wanted keys 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 1 librados: calling monclient init 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 10 monclient: init 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 10 monclient(hunting): picked mon.noname-b con 0x7fe98c009a30 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 10 monclient(hunting): picked mon.noname-c con 0x7fe98c00a540 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 10 monclient(hunting): picked mon.noname-a con 0x7fe98c013510 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00b1870 con 0x7fe98c013510 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00aebf0 con 0x7fe98c009a30 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9a00b09b0 con 0x7fe98c00a540 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.154+0000 7fe9a6182640 10 monclient(hunting): authenticate will time out at 6057.344211s 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.155+0000 7fe99f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe98c009a30 0x7fe9a00ae4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:51350/0 (socket says 172.21.15.32:51350) 2024-03-25T18:39:11.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.155+0000 7fe99f7fe640 1 -- 172.21.15.32:0/3412534476 learned_addr learned my addr 172.21.15.32:0/3412534476 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:11.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.156+0000 7fe987fff640 10 client.?.objecter ms_handle_connect 0x7fe98c009a30 2024-03-25T18:39:11.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.156+0000 7fe987fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.175+0000 7fe987fff640 10 client.?.objecter ms_handle_connect 0x7fe98c013510 2024-03-25T18:39:11.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.175+0000 7fe987fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.179+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142365888 0 0) 0x7fe9a00b1870 con 0x7fe98c013510 2024-03-25T18:39:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.179+0000 7fe987fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.179+0000 7fe987fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.179+0000 7fe987fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.179+0000 7fe987fff640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:39:11.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.179+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe98c00aa10 con 0x7fe98c013510 2024-03-25T18:39:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.187+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3250319911 0 0) 0x7fe9a00aebf0 con 0x7fe98c009a30 2024-03-25T18:39:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.187+0000 7fe987fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.187+0000 7fe987fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.187+0000 7fe987fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.187+0000 7fe987fff640 10 monclient(hunting): my global_id is 4099 2024-03-25T18:39:11.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.187+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9a00b1870 con 0x7fe98c009a30 2024-03-25T18:39:11.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.192+0000 7fe987fff640 10 client.?.objecter ms_handle_connect 0x7fe98c00a540 2024-03-25T18:39:11.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.192+0000 7fe987fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:11.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.194+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3786216517 0 0) 0x7fe98c00aa10 con 0x7fe98c013510 2024-03-25T18:39:11.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.194+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9a00aebf0 con 0x7fe98c013510 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.194+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe998004490 con 0x7fe98c013510 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.194+0000 7fe987fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.194+0000 7fe987fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.194+0000 7fe987fff640 10 monclient(hunting): dump: 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:11.196 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:11.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.194+0000 7fe987fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:11.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 120898012 0 0) 0x7fe9a00aebf0 con 0x7fe98c013510 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 >> v1:172.21.15.32:6790/0 conn(0x7fe98c00a540 legacy=0x7fe9a0138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 >> v1:172.21.15.136:6789/0 conn(0x7fe98c009a30 legacy=0x7fe9a00ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 10 monclient: _finish_hunting 0 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 1 monclient: found mon.a 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9a01cefc0 con 0x7fe98c013510 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 10 monclient: _finish_auth 0 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 10 monclient: _check_auth_tickets 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe987fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 10 monclient: _renew_subs 2024-03-25T18:39:11.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9a00b1300 con 0x7fe98c013510 2024-03-25T18:39:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 10 monclient: _renew_subs 2024-03-25T18:39:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9a01ced00 con 0x7fe98c013510 2024-03-25T18:39:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.240+0000 7fe9a6182640 1 librados: init done 2024-03-25T18:39:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.241+0000 7fe9a6182640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:11.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.241+0000 7fe9a6182640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:11.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.241+0000 7fe9a6182640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:11.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.241+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9a00b14f0 con 0x7fe98c013510 2024-03-25T18:39:11.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9980048a0 con 0x7fe98c013510 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe998004cf0 con 0x7fe98c013510 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 10 monclient: dump: 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:11.443 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fe9980052a0 con 0x7fe98c013510 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe99dffb640 4 set_mon_vals no callback set 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 1499282868 0 0) 0x7fe998006400 con 0x7fe98c013510 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 10 client.4107.objecter ms_dispatch 0x7fe9a0000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fe998006b80 con 0x7fe98c013510 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:11.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.441+0000 7fe987fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.556+0000 7fe9a6182640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:39:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.556+0000 7fe9a6182640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:39:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.556+0000 7fe9a6182640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:11.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.556+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe9a00b14f0 con 0x7fe98c013510 2024-03-25T18:39:11.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.791+0000 7fe987fff640 1 -- 172.21.15.32:0/3412534476 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7fe998005630 con 0x7fe98c013510 2024-03-25T18:39:11.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.791+0000 7fe987fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:39:11.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.791+0000 7fe987fff640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:39:11.792 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 10 librados: watch_flush enter 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 10 librados: watch_flush exit 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 10 monclient: shutdown 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 >> v1:172.21.15.32:6789/0 conn(0x7fe98c013510 legacy=0x7fe9a00aee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 shutdown_connections 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 >> 172.21.15.32:0/3412534476 conn(0x7fe9a00668e0 msgr2=0x7fe9a00b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 shutdown_connections 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 1 -- 172.21.15.32:0/3412534476 wait complete. 2024-03-25T18:39:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:11.792+0000 7fe9a6182640 1 librados: shutdown 2024-03-25T18:39:11.806 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:39:11.807 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:39:11.807 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:39:11.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:39:11.810 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:11.810+0000 7f97b9810040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:39:11.835 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:39:11.835 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:39:11.864 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:11.864+0000 7f97b9810040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:39:11.878 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:11.878+0000 7fe4ca82b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:39:11.904 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:39:11.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:39:11.948 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:11.948+0000 7fe4ca82b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:39:11.971 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-03-25T18:39:11.971 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:39:12.039 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:39:12.039 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:39:12.067 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:39:12.067 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:39:12.135 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:39:12.135 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:39:12.202 DEBUG:teuthology.orchestra.run.smithi136:> set -ex 2024-03-25T18:39:12.202 DEBUG:teuthology.orchestra.run.smithi136:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:39:12.270 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 3c5e9771-b868-42d0-bad2-4d4e060105f4 0 2024-03-25T18:39:12.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.410+0000 7f63952a8640 10 monclient: get_monmap_and_config 2024-03-25T18:39:12.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.410+0000 7f63952a8640 10 monclient: build_initial_monmap 2024-03-25T18:39:12.411 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.410+0000 7f63952a8640 10 monclient: monmap: 2024-03-25T18:39:12.411 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:12.411 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.411 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:39:12.411541+0000 2024-03-25T18:39:12.412 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:39:12.411541+0000 2024-03-25T18:39:12.412 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:12.412 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.412 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:12.412 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:12.412 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:12.412 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.411+0000 7f63952a8640 1 Processor -- start 2024-03-25T18:39:12.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.411+0000 7f63952a8640 1 -- start start 2024-03-25T18:39:12.413 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.411+0000 7f63952a8640 10 monclient: init 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 10 monclient(hunting): picked mon.noname-b con 0x7f63900a9710 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 10 monclient(hunting): picked mon.noname-c con 0x7f63900aafa0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 10 monclient(hunting): picked mon.noname-a con 0x7f639013c570 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63900ab420 con 0x7f639013c570 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63900b0190 con 0x7f63900aafa0 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63900abdb0 con 0x7f63900a9710 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f63952a8640 10 monclient(hunting): authenticate will time out at 6039.577926s 2024-03-25T18:39:12.414 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f6387fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f63900aafa0 0x7f63900724a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38470/0 (socket says 172.21.15.136:38470) 2024-03-25T18:39:12.415 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.412+0000 7f6387fff640 1 -- 172.21.15.136:0/3009798395 learned_addr learned my addr 172.21.15.136:0/3009798395 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:12.415 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.414+0000 7f97b9810040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:39:12.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2545637786 0 0) 0x7f63900b0190 con 0x7f63900aafa0 2024-03-25T18:39:12.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:39:12.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f637c003520 con 0x7f63900aafa0 2024-03-25T18:39:12.416 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3141391979 0 0) 0x7f63900ab420 con 0x7f639013c570 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63900b0190 con 0x7f639013c570 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1531880736 0 0) 0x7f63900abdb0 con 0x7f63900a9710 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:39:12.417 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.415+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63900ab420 con 0x7f63900a9710 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3328874295 0 0) 0x7f637c003520 con 0x7f63900aafa0 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 >> v1:172.21.15.32:6790/0 conn(0x7f63900a9710 legacy=0x7f63900ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 >> v1:172.21.15.32:6789/0 conn(0x7f639013c570 legacy=0x7f639014a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63900abf40 con 0x7f63900aafa0 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6378001690 con 0x7f63900aafa0 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.420 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f638e7fc640 10 monclient: dump: 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f63952a8640 5 monclient: authenticate success, global_id 4108 2024-03-25T18:39:12.421 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.419+0000 7f63952a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63780017f0 con 0x7f63900aafa0 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f638e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f638e7fc640 1 -- 172.21.15.136:0/3009798395 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6378001c00 con 0x7f63900aafa0 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 4 set_mon_vals no callback set 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f638e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f638e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f638e7fc640 10 monclient: dump: 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.423 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 10 monclient: shutdown 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 1 -- 172.21.15.136:0/3009798395 >> v1:172.21.15.136:6789/0 conn(0x7f63900aafa0 legacy=0x7f63900724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 1 -- 172.21.15.136:0/3009798395 shutdown_connections 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 1 -- 172.21.15.136:0/3009798395 >> 172.21.15.136:0/3009798395 conn(0x7f639006b560 msgr2=0x7f63900b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 1 -- 172.21.15.136:0/3009798395 shutdown_connections 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.421+0000 7f63952a8640 1 -- 172.21.15.136:0/3009798395 wait complete. 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.422+0000 7f63952a8640 10 monclient: build_initial_monmap 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.422+0000 7f63952a8640 10 monclient: monmap: 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:12.424 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:12.425 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-25T18:39:12.425 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-25T18:39:12.425 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:12.425 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-25T18:39:12.425 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:12.425 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:12.425 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:12.425 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 librados: starting msgr at 2024-03-25T18:39:12.427 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 librados: starting objecter 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 Processor -- start 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 -- start start 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 librados: setting wanted keys 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 librados: calling monclient init 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 10 monclient: init 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 10 monclient(hunting): picked mon.noname-a con 0x7f63900a9710 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 10 monclient(hunting): picked mon.noname-b con 0x7f63900aafa0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 10 monclient(hunting): picked mon.noname-c con 0x7f639013c570 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639013dd80 con 0x7f63900a9710 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639013d250 con 0x7f63900aafa0 2024-03-25T18:39:12.428 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f639013cf00 con 0x7f639013c570 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.423+0000 7f63952a8640 10 monclient(hunting): authenticate will time out at 6039.589581s 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.424+0000 7f638f7fe640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f639013c570 0x7f63901e2470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.136:34922/0 (socket says 172.21.15.136:34922) 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.424+0000 7f638f7fe640 1 -- 172.21.15.136:0/2882696606 learned_addr learned my addr 172.21.15.136:0/2882696606 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.424+0000 7f63877fe640 10 client.?.objecter ms_handle_connect 0x7f639013c570 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.424+0000 7f63877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.424+0000 7f63877fe640 10 client.?.objecter ms_handle_connect 0x7f63900aafa0 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.424+0000 7f63877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.424+0000 7f63877fe640 10 client.?.objecter ms_handle_connect 0x7f63900a9710 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.424+0000 7f63877fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2239948641 0 0) 0x7f639013cf00 con 0x7f639013c570 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:39:12.429 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6370004eb0 con 0x7f639013c570 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3492122439 0 0) 0x7f639013d250 con 0x7f63900aafa0 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.426+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f639013cf00 con 0x7f63900aafa0 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.427+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53398417 0 0) 0x7f639013dd80 con 0x7f63900a9710 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.427+0000 7f63877fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.427+0000 7f63877fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.427+0000 7f63877fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.427+0000 7f63877fe640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:39:12.430 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.427+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f639013d250 con 0x7f63900a9710 2024-03-25T18:39:12.432 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 306729639 0 0) 0x7f639013d250 con 0x7f63900a9710 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f639013dd80 con 0x7f63900a9710 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6380001830 con 0x7f63900a9710 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 10 monclient(hunting): dump: 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.433 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2243622414 0 0) 0x7f639013cf00 con 0x7f63900aafa0 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f639013d250 con 0x7f63900aafa0 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6378001b00 con 0x7f63900aafa0 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.432+0000 7f63877fe640 10 monclient(hunting): dump: 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.434 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.435 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.435 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.435 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.435 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.436 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.435+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3217317211 0 0) 0x7f639013dd80 con 0x7f63900a9710 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 >> v1:172.21.15.32:6790/0 conn(0x7f639013c570 legacy=0x7f63901e2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 >> v1:172.21.15.136:6789/0 conn(0x7f63900aafa0 legacy=0x7f639013d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 10 monclient: _finish_hunting 0 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 1 monclient: found mon.a 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f639013f6c0 con 0x7f63900a9710 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 10 monclient: _finish_auth 0 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 10 monclient: _check_auth_tickets 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63877fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 10 monclient: _renew_subs 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:12.437 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f639013f970 con 0x7f63900a9710 2024-03-25T18:39:12.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:12.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 10 monclient: _renew_subs 2024-03-25T18:39:12.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:12.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6390138720 con 0x7f63900a9710 2024-03-25T18:39:12.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 1 librados: init done 2024-03-25T18:39:12.438 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:12.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:12.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:12.439 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.436+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f639013d090 con 0x7f63900a9710 2024-03-25T18:39:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6380001d40 con 0x7f63900a9710 2024-03-25T18:39:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f63800037b0 con 0x7f63900a9710 2024-03-25T18:39:12.443 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 10 monclient: dump: 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f638dffb640 4 set_mon_vals no callback set 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6380003ce0 con 0x7f63900a9710 2024-03-25T18:39:12.444 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 955873826 0 0) 0x7f6380005180 con 0x7f63900a9710 2024-03-25T18:39:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 10 client.4119.objecter ms_dispatch 0x7f6390000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:39:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:39:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:39:12.445 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.442+0000 7f63877fe640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:39:12.473 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.472+0000 7f97b9810040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:39:12.515 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:12.515+0000 7fe4ca82b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:39:12.517 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.516+0000 7f97b9810040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:39:12.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.541+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6380005560 con 0x7f63900a9710 2024-03-25T18:39:12.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.541+0000 7f63877fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:12.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.541+0000 7f63877fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:12.571 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.571+0000 7f97b9810040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:39:12.571 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:12.571+0000 7fe4ca82b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:39:12.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.614+0000 7f63952a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3c5e9771-b868-42d0-bad2-4d4e060105f4", "id": 0}] 2024-03-25T18:39:12.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.614+0000 7f63952a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3c5e9771-b868-42d0-bad2-4d4e060105f4", "id": 0}] 2024-03-25T18:39:12.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.614+0000 7f63952a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:12.615 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.614+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3c5e9771-b868-42d0-bad2-4d4e060105f4", "id": 0} v 0) v1 -- 0x7f639004f730 con 0x7f63900a9710 2024-03-25T18:39:12.615 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.614+0000 7f97b9810040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:39:12.618 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:12.618+0000 7fe4ca82b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:39:12.674 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:12.674+0000 7fe4ca82b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:39:12.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.699+0000 7f63877fe640 1 -- 172.21.15.136:0/2882696606 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3c5e9771-b868-42d0-bad2-4d4e060105f4", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3014440503 0 732707515) 0x7f63800057b0 con 0x7f63900a9710 2024-03-25T18:39:12.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.699+0000 7f63877fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3c5e9771-b868-42d0-bad2-4d4e060105f4", "id": 0}] 2024-03-25T18:39:12.700 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.699+0000 7f63877fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:12.700 INFO:teuthology.orchestra.run.smithi136.stdout:0 2024-03-25T18:39:12.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 10 librados: watch_flush enter 2024-03-25T18:39:12.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 10 librados: watch_flush exit 2024-03-25T18:39:12.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:39:12.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:39:12.701 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 10 monclient: shutdown 2024-03-25T18:39:12.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:12.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 >> v1:172.21.15.32:6789/0 conn(0x7f63900a9710 legacy=0x7f6390141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 shutdown_connections 2024-03-25T18:39:12.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 >> 172.21.15.136:0/2882696606 conn(0x7f639006b560 msgr2=0x7f6390142730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:12.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 shutdown_connections 2024-03-25T18:39:12.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 1 -- 172.21.15.136:0/2882696606 wait complete. 2024-03-25T18:39:12.702 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.700+0000 7f63952a8640 1 librados: shutdown 2024-03-25T18:39:12.712 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 264b5aa3-cfe9-4b37-b248-601a91ea8907 1 2024-03-25T18:39:12.719 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:12.720+0000 7fe4ca82b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:39:12.733 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.733+0000 7f97b9810040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:39:12.808 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.808+0000 7f97b9810040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:39:12.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.810+0000 7fec57493640 10 monclient: get_monmap_and_config 2024-03-25T18:39:12.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.810+0000 7fec57493640 10 monclient: build_initial_monmap 2024-03-25T18:39:12.811 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.810+0000 7fec57493640 10 monclient: monmap: 2024-03-25T18:39:12.811 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:12.811 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.811 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:39:12.811647+0000 2024-03-25T18:39:12.811 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:39:12.811647+0000 2024-03-25T18:39:12.811 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:12.812 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.812 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:12.812 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:12.812 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:12.812 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.811+0000 7fec57493640 1 Processor -- start 2024-03-25T18:39:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.811+0000 7fec57493640 1 -- start start 2024-03-25T18:39:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.811+0000 7fec57493640 10 monclient: init 2024-03-25T18:39:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:39:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 10 monclient(hunting): picked mon.noname-c con 0x7fec50070210 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:12.813 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 10 monclient(hunting): picked mon.noname-b con 0x7fec500b06d0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 10 monclient(hunting): picked mon.noname-a con 0x7fec500705f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec500b0b80 con 0x7fec500705f0 2024-03-25T18:39:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec500b0430 con 0x7fec50070210 2024-03-25T18:39:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec50071670 con 0x7fec500b06d0 2024-03-25T18:39:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec57493640 10 monclient(hunting): authenticate will time out at 6039.978017s 2024-03-25T18:39:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec5522f640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fec50070210 0x7fec50140930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38488/0 (socket says 172.21.15.136:38488) 2024-03-25T18:39:12.814 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.812+0000 7fec5522f640 1 -- 172.21.15.136:0/292863703 learned_addr learned my addr 172.21.15.136:0/292863703 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:12.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 897256089 0 0) 0x7fec4c001110 con 0x7fec50070210 2024-03-25T18:39:12.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.816 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec3c003520 con 0x7fec50070210 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 40593432 0 0) 0x7fec50071670 con 0x7fec500b06d0 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec4c001110 con 0x7fec500b06d0 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 692527684 0 0) 0x7fec500b0b80 con 0x7fec500705f0 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:39:12.817 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.815+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec50071670 con 0x7fec500705f0 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1755358315 0 0) 0x7fec3c003520 con 0x7fec50070210 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 >> v1:172.21.15.32:6790/0 conn(0x7fec500b06d0 legacy=0x7fec500afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 >> v1:172.21.15.32:6789/0 conn(0x7fec500705f0 legacy=0x7fec5014b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 10 monclient: _finish_hunting 0 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 1 monclient: found mon.noname-c 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50070a60 con 0x7fec50070210 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 10 monclient: _finish_auth 0 2024-03-25T18:39:12.820 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 10 monclient: _check_auth_tickets 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fec4c001680 con 0x7fec50070210 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec57493640 5 monclient: authenticate success, global_id 4114 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec57493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec47fff640 10 monclient: dump: 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.821 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.822 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.822 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.819+0000 7fec57493640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec4c0017e0 con 0x7fec50070210 2024-03-25T18:39:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec47fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:12.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec47fff640 1 -- 172.21.15.136:0/292863703 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fec4c001b70 con 0x7fec50070210 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec47fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec47fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec47fff640 10 monclient: dump: 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.825 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec57493640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec57493640 4 set_mon_vals no callback set 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec57493640 10 monclient: shutdown 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec57493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.821+0000 7fec57493640 1 -- 172.21.15.136:0/292863703 >> v1:172.21.15.136:6789/0 conn(0x7fec50070210 legacy=0x7fec50140930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.822+0000 7fec57493640 1 -- 172.21.15.136:0/292863703 shutdown_connections 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.822+0000 7fec57493640 1 -- 172.21.15.136:0/292863703 >> 172.21.15.136:0/292863703 conn(0x7fec500b5cd0 msgr2=0x7fec500b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.822+0000 7fec57493640 1 -- 172.21.15.136:0/292863703 shutdown_connections 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.822+0000 7fec57493640 1 -- 172.21.15.136:0/292863703 wait complete. 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.822+0000 7fec57493640 10 monclient: build_initial_monmap 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.822+0000 7fec57493640 10 monclient: monmap: 2024-03-25T18:39:12.826 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 librados: starting msgr at 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 librados: starting objecter 2024-03-25T18:39:12.827 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 Processor -- start 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 -- start start 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 librados: setting wanted keys 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 librados: calling monclient init 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 10 monclient: init 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 10 monclient(hunting): picked mon.noname-b con 0x7fec50070210 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 10 monclient(hunting): picked mon.noname-c con 0x7fec500705f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 10 monclient(hunting): picked mon.noname-a con 0x7fec500b06d0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5006ce10 con 0x7fec500b06d0 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5006d950 con 0x7fec50070210 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec5006dae0 con 0x7fec500705f0 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.823+0000 7fec57493640 10 monclient(hunting): authenticate will time out at 6039.989712s 2024-03-25T18:39:12.828 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.824+0000 7fec5522f640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fec50070210 0x7fec5006c5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38502/0 (socket says 172.21.15.136:38502) 2024-03-25T18:39:12.829 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.824+0000 7fec5522f640 1 -- 172.21.15.136:0/690999749 learned_addr learned my addr 172.21.15.136:0/690999749 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.824+0000 7fec45ffb640 10 client.?.objecter ms_handle_connect 0x7fec50070210 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.824+0000 7fec45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.824+0000 7fec45ffb640 10 client.?.objecter ms_handle_connect 0x7fec500705f0 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.824+0000 7fec45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.824+0000 7fec45ffb640 10 client.?.objecter ms_handle_connect 0x7fec500b06d0 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.824+0000 7fec45ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3736045546 0 0) 0x7fec5006d950 con 0x7fec50070210 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.830 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec34004eb0 con 0x7fec50070210 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4276979090 0 0) 0x7fec5006dae0 con 0x7fec500705f0 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.826+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec5006d950 con 0x7fec500705f0 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.827+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2985962149 0 0) 0x7fec5006ce10 con 0x7fec500b06d0 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.827+0000 7fec45ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.827+0000 7fec45ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.827+0000 7fec45ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.827+0000 7fec45ffb640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:39:12.831 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.827+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec5006dae0 con 0x7fec500b06d0 2024-03-25T18:39:12.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3343440622 0 0) 0x7fec34004eb0 con 0x7fec50070210 2024-03-25T18:39:12.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec5006ce10 con 0x7fec50070210 2024-03-25T18:39:12.832 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fec4c0047e0 con 0x7fec50070210 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:12.833 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 747297961 0 0) 0x7fec5006d950 con 0x7fec500705f0 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec34004eb0 con 0x7fec500705f0 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fec40001910 con 0x7fec500705f0 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.831+0000 7fec45ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.834 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.832+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1475983721 0 0) 0x7fec5006dae0 con 0x7fec500b06d0 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.832+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec5006d950 con 0x7fec500b06d0 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.832+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fec48001c70 con 0x7fec500b06d0 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.832+0000 7fec45ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.832+0000 7fec45ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.832+0000 7fec45ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.836 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 416452556 0 0) 0x7fec5006ce10 con 0x7fec50070210 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 >> v1:172.21.15.32:6790/0 conn(0x7fec500705f0 legacy=0x7fec50139820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 >> v1:172.21.15.32:6789/0 conn(0x7fec500b06d0 legacy=0x7fec501daa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 1 monclient: found mon.b 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:12.837 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec50139f30 con 0x7fec50070210 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 10 monclient: _finish_auth 0 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec45ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 5 monclient: authenticate success, global_id 4117 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 10 monclient: _renew_subs 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec5013a1a0 con 0x7fec50070210 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 10 monclient: _renew_subs 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec501363b0 con 0x7fec50070210 2024-03-25T18:39:12.838 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.834+0000 7fec57493640 1 librados: init done 2024-03-25T18:39:12.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.835+0000 7fec57493640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:12.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.835+0000 7fec57493640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:12.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.835+0000 7fec57493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:12.839 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.835+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec5004f730 con 0x7fec50070210 2024-03-25T18:39:12.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec4c001360 con 0x7fec50070210 2024-03-25T18:39:12.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:12.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fec4c004bf0 con 0x7fec50070210 2024-03-25T18:39:12.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:12.840 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 10 monclient: dump: 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:12.841 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:12.842 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fec4c005120 con 0x7fec50070210 2024-03-25T18:39:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec477fe640 4 set_mon_vals no callback set 2024-03-25T18:39:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 2063067987 0 0) 0x7fec4c006630 con 0x7fec50070210 2024-03-25T18:39:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 10 client.4117.objecter ms_dispatch 0x7fec50000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:39:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:39:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:39:12.842 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:12.839+0000 7fec45ffb640 20 client.4117.objecter dump_active .. 0 homeless 2024-03-25T18:39:12.846 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:12.846+0000 7fe4ca82b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:39:12.859 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.859+0000 7f97b9810040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:39:12.913 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.913+0000 7f97b9810040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:39:12.919 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:12.919+0000 7fe4ca82b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:39:12.962 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:12.962+0000 7f97b9810040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:39:12.967 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:12.967+0000 7fe4ca82b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:39:13.011 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.011+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fec4c006a10 con 0x7fec50070210 2024-03-25T18:39:13.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.011+0000 7fec45ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:13.012 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.011+0000 7fec45ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:13.025 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:13.025+0000 7fe4ca82b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:39:13.081 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:13.081+0000 7fe4ca82b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:39:13.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.090+0000 7fec57493640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "264b5aa3-cfe9-4b37-b248-601a91ea8907", "id": 1}] 2024-03-25T18:39:13.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.090+0000 7fec57493640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "264b5aa3-cfe9-4b37-b248-601a91ea8907", "id": 1}] 2024-03-25T18:39:13.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.090+0000 7fec57493640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.091 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.090+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "264b5aa3-cfe9-4b37-b248-601a91ea8907", "id": 1} v 0) v1 -- 0x7fec5013ff60 con 0x7fec50070210 2024-03-25T18:39:13.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.214+0000 7fec45ffb640 1 -- 172.21.15.136:0/690999749 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "264b5aa3-cfe9-4b37-b248-601a91ea8907", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2739369246 0 940483276) 0x7fec4c0054a0 con 0x7fec50070210 2024-03-25T18:39:13.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.214+0000 7fec45ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "264b5aa3-cfe9-4b37-b248-601a91ea8907", "id": 1}] 2024-03-25T18:39:13.215 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.214+0000 7fec45ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:13.215 INFO:teuthology.orchestra.run.smithi136.stdout:1 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 10 librados: watch_flush enter 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 10 librados: watch_flush exit 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 20 client.4117.objecter shutdown clearing up homeless session... 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 10 client.4117.objecter successfully canceled tick 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 10 monclient: shutdown 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 >> v1:172.21.15.136:6789/0 conn(0x7fec50070210 legacy=0x7fec5006c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 shutdown_connections 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 >> 172.21.15.136:0/690999749 conn(0x7fec500b5cd0 msgr2=0x7fec500afc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 shutdown_connections 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 1 -- 172.21.15.136:0/690999749 wait complete. 2024-03-25T18:39:13.217 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.215+0000 7fec57493640 1 librados: shutdown 2024-03-25T18:39:13.228 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 7d429f95-7ad6-426f-bc51-b9e2a750a918 2 2024-03-25T18:39:13.365 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:13.364+0000 7f97b9810040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:39:13.368 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.368+0000 7fefd8825640 10 monclient: get_monmap_and_config 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.368+0000 7fefd8825640 10 monclient: build_initial_monmap 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.368+0000 7fefd8825640 10 monclient: monmap: 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:39:13.368948+0000 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:39:13.368948+0000 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.368+0000 7fefd8825640 1 Processor -- start 2024-03-25T18:39:13.369 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.368+0000 7fefd8825640 1 -- start start 2024-03-25T18:39:13.370 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.368+0000 7fefd8825640 10 monclient: init 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 10 monclient(hunting): picked mon.noname-b con 0x7fefd40b4700 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 10 monclient(hunting): picked mon.noname-c con 0x7fefd4142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 10 monclient(hunting): picked mon.noname-a con 0x7fefd40a96b0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd40a9d60 con 0x7fefd40a96b0 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd413ff10 con 0x7fefd4142a20 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd40aa1f0 con 0x7fefd40b4700 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:13.371 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.369+0000 7fefd8825640 10 monclient(hunting): authenticate will time out at 6040.535352s 2024-03-25T18:39:13.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.370+0000 7fefd1d9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fefd4142a20 0x7fefd413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38516/0 (socket says 172.21.15.136:38516) 2024-03-25T18:39:13.372 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.370+0000 7fefd1d9b640 1 -- 172.21.15.136:0/732979983 learned_addr learned my addr 172.21.15.136:0/732979983 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.372+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819289484 0 0) 0x7fefd413ff10 con 0x7fefd4142a20 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.372+0000 7fefd159a640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.372+0000 7fefd159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.372+0000 7fefd159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.372+0000 7fefd159a640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.372+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefbc003520 con 0x7fefd4142a20 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581174402 0 0) 0x7fefd40a9d60 con 0x7fefd40a96b0 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.374 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:39:13.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd413ff10 con 0x7fefd40a96b0 2024-03-25T18:39:13.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 119548080 0 0) 0x7fefd40aa1f0 con 0x7fefd40b4700 2024-03-25T18:39:13.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:39:13.375 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.373+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd40a9d60 con 0x7fefd40b4700 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2150346877 0 0) 0x7fefbc003520 con 0x7fefd4142a20 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 >> v1:172.21.15.32:6790/0 conn(0x7fefd40b4700 legacy=0x7fefd4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 >> v1:172.21.15.32:6789/0 conn(0x7fefd40a96b0 legacy=0x7fefd413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 10 monclient: _finish_hunting 0 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 1 monclient: found mon.noname-c 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd40b4ae0 con 0x7fefd4142a20 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 10 monclient: _finish_auth 0 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 10 monclient: _check_auth_tickets 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fefc4001690 con 0x7fefd4142a20 2024-03-25T18:39:13.377 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd159a640 10 monclient: dump: 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd8825640 5 monclient: authenticate success, global_id 4120 2024-03-25T18:39:13.378 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.376+0000 7fefd8825640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:13.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.378+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefc40017f0 con 0x7fefd4142a20 2024-03-25T18:39:13.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.378+0000 7fefd159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:13.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.378+0000 7fefd159a640 1 -- 172.21.15.136:0/732979983 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fefc4001c00 con 0x7fefd4142a20 2024-03-25T18:39:13.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.378+0000 7fefd159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.378+0000 7fefd159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:13.379 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.378+0000 7fefd159a640 10 monclient: dump: 2024-03-25T18:39:13.379 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.378+0000 7fefd8825640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:13.380 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.378+0000 7fefd8825640 4 set_mon_vals no callback set 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.379+0000 7fefd8825640 10 monclient: shutdown 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.379+0000 7fefd8825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.379+0000 7fefd8825640 1 -- 172.21.15.136:0/732979983 >> v1:172.21.15.136:6789/0 conn(0x7fefd4142a20 legacy=0x7fefd413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.379+0000 7fefd8825640 1 -- 172.21.15.136:0/732979983 shutdown_connections 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.379+0000 7fefd8825640 1 -- 172.21.15.136:0/732979983 >> 172.21.15.136:0/732979983 conn(0x7fefd40668e0 msgr2=0x7fefd4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.379+0000 7fefd8825640 1 -- 172.21.15.136:0/732979983 shutdown_connections 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.379+0000 7fefd8825640 1 -- 172.21.15.136:0/732979983 wait complete. 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 10 monclient: build_initial_monmap 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 10 monclient: monmap: 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:13.383 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 1 librados: starting msgr at 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 1 librados: starting objecter 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 1 Processor -- start 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 1 -- start start 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 1 librados: setting wanted keys 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 1 librados: calling monclient init 2024-03-25T18:39:13.384 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.380+0000 7fefd8825640 10 monclient: init 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 10 monclient(hunting): picked mon.noname-b con 0x7fefd40a96b0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 10 monclient(hunting): picked mon.noname-c con 0x7fefd40b4700 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 10 monclient(hunting): picked mon.noname-a con 0x7fefd4142a20 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd41d7590 con 0x7fefd4142a20 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd413ccc0 con 0x7fefd40a96b0 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefd41d8180 con 0x7fefd40b4700 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd8825640 10 monclient(hunting): authenticate will time out at 6040.547144s 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd259c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fefd40a96b0 0x7fefd41d7a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38532/0 (socket says 172.21.15.136:38532) 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.381+0000 7fefd259c640 1 -- 172.21.15.136:0/1371706183 learned_addr learned my addr 172.21.15.136:0/1371706183 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.382+0000 7fefc2ffd640 10 client.?.objecter ms_handle_connect 0x7fefd40a96b0 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.382+0000 7fefc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:13.385 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.382+0000 7fefc2ffd640 10 client.?.objecter ms_handle_connect 0x7fefd40b4700 2024-03-25T18:39:13.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.382+0000 7fefc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:13.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.382+0000 7fefc2ffd640 10 client.?.objecter ms_handle_connect 0x7fefd4142a20 2024-03-25T18:39:13.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.382+0000 7fefc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993517068 0 0) 0x7fefd413ccc0 con 0x7fefd40a96b0 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefb4004eb0 con 0x7fefd40a96b0 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1855698385 0 0) 0x7fefd41d7590 con 0x7fefd4142a20 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd413ccc0 con 0x7fefd4142a20 2024-03-25T18:39:13.387 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3914151340 0 0) 0x7fefd41d8180 con 0x7fefd40b4700 2024-03-25T18:39:13.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:39:13.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.384+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd41d7590 con 0x7fefd40b4700 2024-03-25T18:39:13.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.389+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3761945301 0 0) 0x7fefb4004eb0 con 0x7fefd40a96b0 2024-03-25T18:39:13.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.389+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefd41d8180 con 0x7fefd40a96b0 2024-03-25T18:39:13.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.389+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fefc80019b0 con 0x7fefd40a96b0 2024-03-25T18:39:13.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.389+0000 7fefc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.389+0000 7fefc2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:13.390 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.389+0000 7fefc2ffd640 10 monclient(hunting): dump: 2024-03-25T18:39:13.390 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.390 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.391 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.389+0000 7fefc2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3968079497 0 0) 0x7fefd41d7590 con 0x7fefd40b4700 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefb4004eb0 con 0x7fefd40b4700 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fefc4001b00 con 0x7fefd40b4700 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 10 monclient(hunting): dump: 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.394 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1539754496 0 0) 0x7fefd41d8180 con 0x7fefd40a96b0 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 >> v1:172.21.15.32:6790/0 conn(0x7fefd40b4700 legacy=0x7fefd41d4650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 >> v1:172.21.15.32:6789/0 conn(0x7fefd4142a20 legacy=0x7fefd413b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 1 monclient: found mon.b 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefd41d8310 con 0x7fefd40a96b0 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:39:13.395 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 10 monclient: _renew_subs 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefd41d3fd0 con 0x7fefd40a96b0 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 10 monclient: _renew_subs 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefd41d4f40 con 0x7fefd40a96b0 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.392+0000 7fefd8825640 1 librados: init done 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.393+0000 7fefd8825640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.393+0000 7fefd8825640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.393+0000 7fefd8825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.396 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.393+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefd404f730 con 0x7fefd40a96b0 2024-03-25T18:39:13.490 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:13.490+0000 7fe4ca82b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:39:13.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefc8003610 con 0x7fefd40a96b0 2024-03-25T18:39:13.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:13.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fefc8003920 con 0x7fefd40a96b0 2024-03-25T18:39:13.492 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 10 monclient: dump: 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fefc8003ed0 con 0x7fefd40a96b0 2024-03-25T18:39:13.493 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefd0d99640 4 set_mon_vals no callback set 2024-03-25T18:39:13.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1871718201 0 0) 0x7fefc8006670 con 0x7fefd40a96b0 2024-03-25T18:39:13.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 10 client.4123.objecter ms_dispatch 0x7fefd4000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:39:13.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:39:13.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:39:13.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-25T18:39:13.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fefc8006ab0 con 0x7fefd40a96b0 2024-03-25T18:39:13.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:13.494 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.491+0000 7fefc2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:13.509 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:13.509+0000 7f97b9810040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:39:13.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.565+0000 7fefd8825640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "7d429f95-7ad6-426f-bc51-b9e2a750a918", "id": 2}] 2024-03-25T18:39:13.565 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.565+0000 7fefd8825640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "7d429f95-7ad6-426f-bc51-b9e2a750a918", "id": 2}] 2024-03-25T18:39:13.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.565+0000 7fefd8825640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.566 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.565+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "7d429f95-7ad6-426f-bc51-b9e2a750a918", "id": 2} v 0) v1 -- 0x7fefd4069f30 con 0x7fefd40a96b0 2024-03-25T18:39:13.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.636+0000 7fefc2ffd640 1 -- 172.21.15.136:0/1371706183 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "7d429f95-7ad6-426f-bc51-b9e2a750a918", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 129867414 0 216599125) 0x7fefc8004250 con 0x7fefd40a96b0 2024-03-25T18:39:13.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.636+0000 7fefc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "7d429f95-7ad6-426f-bc51-b9e2a750a918", "id": 2}] 2024-03-25T18:39:13.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.636+0000 7fefc2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:13.637 INFO:teuthology.orchestra.run.smithi136.stdout:2 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 10 librados: watch_flush enter 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 10 librados: watch_flush exit 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 10 client.4123.objecter successfully canceled tick 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 10 monclient: shutdown 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 >> v1:172.21.15.136:6789/0 conn(0x7fefd40a96b0 legacy=0x7fefd41d7a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 shutdown_connections 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 >> 172.21.15.136:0/1371706183 conn(0x7fefd40668e0 msgr2=0x7fefd40b26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:13.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 shutdown_connections 2024-03-25T18:39:13.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 1 -- 172.21.15.136:0/1371706183 wait complete. 2024-03-25T18:39:13.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.637+0000 7fefd8825640 1 librados: shutdown 2024-03-25T18:39:13.641 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:13.641+0000 7f97b9810040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:39:13.643 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:13.643+0000 7fe4ca82b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:39:13.648 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 9fa2130c-1583-40e1-97c5-dc72314a5358 3 2024-03-25T18:39:13.690 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:13.690+0000 7f97b9810040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:39:13.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.746+0000 7f166a75d640 10 monclient: get_monmap_and_config 2024-03-25T18:39:13.746 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.746+0000 7f166a75d640 10 monclient: build_initial_monmap 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.746+0000 7f166a75d640 10 monclient: monmap: 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:39:13.746973+0000 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:39:13.746973+0000 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:13.747 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.746+0000 7f166a75d640 1 Processor -- start 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.746+0000 7f166a75d640 1 -- start start 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.746+0000 7f166a75d640 10 monclient: init 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 10 monclient(hunting): picked mon.noname-b con 0x7f16640b4700 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 10 monclient(hunting): picked mon.noname-a con 0x7f1664142a20 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 10 monclient(hunting): picked mon.noname-c con 0x7f16640a96b0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640a9d60 con 0x7f1664142a20 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f166413ff10 con 0x7f16640a96b0 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640aa1f0 con 0x7f16640b4700 2024-03-25T18:39:13.749 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:13.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.747+0000 7f166a75d640 10 monclient(hunting): authenticate will time out at 6040.913304s 2024-03-25T18:39:13.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.748+0000 7f1668cfa640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f16640a96b0 0x7f166413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38534/0 (socket says 172.21.15.136:38534) 2024-03-25T18:39:13.750 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.748+0000 7f1668cfa640 1 -- 172.21.15.136:0/1610578150 learned_addr learned my addr 172.21.15.136:0/1610578150 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:13.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.750+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3371780627 0 0) 0x7f16640aa1f0 con 0x7f16640b4700 2024-03-25T18:39:13.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.750+0000 7f1662ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.750+0000 7f1662ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.750+0000 7f1662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.750+0000 7f1662ffd640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:39:13.751 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.750+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f164c003520 con 0x7f16640b4700 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3795224684 0 0) 0x7f16640a9d60 con 0x7f1664142a20 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16640aa1f0 con 0x7f1664142a20 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3673203201 0 0) 0x7f166413ff10 con 0x7f16640a96b0 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:39:13.752 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.751+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16640a9d60 con 0x7f16640a96b0 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1920344468 0 0) 0x7f164c003520 con 0x7f16640b4700 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 >> v1:172.21.15.136:6789/0 conn(0x7f16640a96b0 legacy=0x7f166413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 >> v1:172.21.15.32:6789/0 conn(0x7f1664142a20 legacy=0x7f166413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 1 monclient: found mon.noname-b 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640b4ae0 con 0x7f16640b4700 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 10 monclient: _finish_auth 0 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:39:13.755 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f166a75d640 5 monclient: authenticate success, global_id 4133 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1654001690 con 0x7f16640b4700 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f166a75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f1662ffd640 10 monclient: dump: 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.756 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.754+0000 7f166a75d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:13.757 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:13.755+0000 7fe4ca82b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:39:13.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.756+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16540017f0 con 0x7f16640b4700 2024-03-25T18:39:13.757 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.756+0000 7f1662ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.756+0000 7f1662ffd640 1 -- 172.21.15.136:0/1610578150 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1654001c00 con 0x7f16640b4700 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.756+0000 7f1662ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.756+0000 7f1662ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.756+0000 7f1662ffd640 10 monclient: dump: 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.758 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.756+0000 7f166a75d640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:13.759 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.756+0000 7f166a75d640 4 set_mon_vals no callback set 2024-03-25T18:39:13.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.757+0000 7f166a75d640 10 monclient: shutdown 2024-03-25T18:39:13.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.757+0000 7f166a75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:13.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.757+0000 7f166a75d640 1 -- 172.21.15.136:0/1610578150 >> v1:172.21.15.32:6790/0 conn(0x7f16640b4700 legacy=0x7f1664142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.757+0000 7f166a75d640 1 -- 172.21.15.136:0/1610578150 shutdown_connections 2024-03-25T18:39:13.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.757+0000 7f166a75d640 1 -- 172.21.15.136:0/1610578150 >> 172.21.15.136:0/1610578150 conn(0x7f16640668e0 msgr2=0x7f1664066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:13.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.757+0000 7f166a75d640 1 -- 172.21.15.136:0/1610578150 shutdown_connections 2024-03-25T18:39:13.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.757+0000 7f166a75d640 1 -- 172.21.15.136:0/1610578150 wait complete. 2024-03-25T18:39:13.761 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.758+0000 7f166a75d640 10 monclient: build_initial_monmap 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.758+0000 7f166a75d640 10 monclient: monmap: 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.759+0000 7f166a75d640 1 librados: starting msgr at 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.759+0000 7f166a75d640 1 librados: starting objecter 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.759+0000 7f166a75d640 1 Processor -- start 2024-03-25T18:39:13.762 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.759+0000 7f166a75d640 1 -- start start 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.759+0000 7f166a75d640 1 librados: setting wanted keys 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.759+0000 7f166a75d640 1 librados: calling monclient init 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.759+0000 7f166a75d640 10 monclient: init 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 10 monclient(hunting): picked mon.noname-b con 0x7f16640a96b0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 10 monclient(hunting): picked mon.noname-a con 0x7f16640b4700 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 10 monclient(hunting): picked mon.noname-c con 0x7f1664142a20 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640adde0 con 0x7f16640b4700 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640ae0c0 con 0x7f16640a96b0 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16640ace10 con 0x7f1664142a20 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f166a75d640 10 monclient(hunting): authenticate will time out at 6040.925900s 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f16637fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f16640b4700 0x7f16640ae4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.136:52510/0 (socket says 172.21.15.136:52510) 2024-03-25T18:39:13.763 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f1668cfa640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f1664142a20 0x7f16640ac1f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.136:34984/0 (socket says 172.21.15.136:34984) 2024-03-25T18:39:13.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f16637fe640 1 -- 172.21.15.136:0/944506078 learned_addr learned my addr 172.21.15.136:0/944506078 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:13.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f1660ff9640 10 client.?.objecter ms_handle_connect 0x7f1664142a20 2024-03-25T18:39:13.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f1660ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:13.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f1660ff9640 10 client.?.objecter ms_handle_connect 0x7f16640b4700 2024-03-25T18:39:13.764 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.760+0000 7f1660ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.761+0000 7f1660ff9640 10 client.?.objecter ms_handle_connect 0x7f16640a96b0 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.761+0000 7f1660ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3651606049 0 0) 0x7f16640ace10 con 0x7f1664142a20 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1650004eb0 con 0x7f1664142a20 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527173575 0 0) 0x7f16640adde0 con 0x7f16640b4700 2024-03-25T18:39:13.765 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16640ace10 con 0x7f16640b4700 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169799814 0 0) 0x7f16640ae0c0 con 0x7f16640a96b0 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:39:13.766 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.763+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16640adde0 con 0x7f16640a96b0 2024-03-25T18:39:13.768 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.767+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388914227 0 0) 0x7f16640ace10 con 0x7f16640b4700 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.767+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16640ae0c0 con 0x7f16640b4700 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1390668256 0 0) 0x7f16640adde0 con 0x7f16640a96b0 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16640ace10 con 0x7f16640a96b0 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1654001b00 con 0x7f16640a96b0 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 10 monclient(hunting): dump: 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.769 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1658001a50 con 0x7f16640b4700 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.768+0000 7f1660ff9640 10 monclient(hunting): dump: 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.770 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.771 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.771 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.771 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.771 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.771 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.771 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.771 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.770+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1732085169 0 0) 0x7f1650004eb0 con 0x7f1664142a20 2024-03-25T18:39:13.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.770+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16640adde0 con 0x7f1664142a20 2024-03-25T18:39:13.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.770+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f165c001bd0 con 0x7f1664142a20 2024-03-25T18:39:13.772 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.770+0000 7f1660ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.770+0000 7f1660ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.770+0000 7f1660ff9640 10 monclient(hunting): dump: 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2895912111 0 0) 0x7f16640ace10 con 0x7f16640a96b0 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 >> v1:172.21.15.32:6790/0 conn(0x7f1664142a20 legacy=0x7f16640ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 >> v1:172.21.15.32:6789/0 conn(0x7f16640b4700 legacy=0x7f16640ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:39:13.773 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 1 monclient: found mon.b 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16640aba80 con 0x7f16640a96b0 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 10 monclient: _finish_auth 0 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f1660ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 10 monclient: _renew_subs 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16640ac900 con 0x7f16640a96b0 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 10 monclient: _renew_subs 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16641e3480 con 0x7f16640a96b0 2024-03-25T18:39:13.774 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.771+0000 7f166a75d640 1 librados: init done 2024-03-25T18:39:13.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.772+0000 7f166a75d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:13.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.772+0000 7f166a75d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:13.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.772+0000 7f166a75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.775 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.772+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16640acfa0 con 0x7f16640a96b0 2024-03-25T18:39:13.804 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:13.804+0000 7fe4ca82b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:39:13.809 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:13.809+0000 7f97b9810040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:39:13.870 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1654001c60 con 0x7f16640a96b0 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1654003a40 con 0x7f16640a96b0 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 10 monclient: dump: 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:13.871 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1654003f30 con 0x7f16640a96b0 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f16627fc640 4 set_mon_vals no callback set 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 3243836559 0 0) 0x7f1654005420 con 0x7f16640a96b0 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 10 client.4129.objecter ms_dispatch 0x7f1664000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f16540057e0 con 0x7f16640a96b0 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:13.872 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.869+0000 7f1660ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:13.906 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:13.906+0000 7f97b9810040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:39:13.921 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:13.921+0000 7fe4ca82b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:39:13.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.941+0000 7f166a75d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9fa2130c-1583-40e1-97c5-dc72314a5358", "id": 3}] 2024-03-25T18:39:13.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.941+0000 7f166a75d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9fa2130c-1583-40e1-97c5-dc72314a5358", "id": 3}] 2024-03-25T18:39:13.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.941+0000 7f166a75d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:13.942 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.941+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9fa2130c-1583-40e1-97c5-dc72314a5358", "id": 3} v 0) v1 -- 0x7f166404f730 con 0x7f16640a96b0 2024-03-25T18:39:13.999 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.999+0000 7f1660ff9640 1 -- 172.21.15.136:0/944506078 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9fa2130c-1583-40e1-97c5-dc72314a5358", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 617109736 0 525046306) 0x7f16540050c0 con 0x7f16640a96b0 2024-03-25T18:39:14.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.999+0000 7f1660ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9fa2130c-1583-40e1-97c5-dc72314a5358", "id": 3}] 2024-03-25T18:39:14.000 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:13.999+0000 7f1660ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:14.000 INFO:teuthology.orchestra.run.smithi136.stdout:3 2024-03-25T18:39:14.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 10 librados: watch_flush enter 2024-03-25T18:39:14.001 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 10 librados: watch_flush exit 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 10 client.4129.objecter successfully canceled tick 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 10 monclient: shutdown 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 >> v1:172.21.15.136:6789/0 conn(0x7f16640a96b0 legacy=0x7f166413a7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 shutdown_connections 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 >> 172.21.15.136:0/944506078 conn(0x7f16640668e0 msgr2=0x7f16640b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 shutdown_connections 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 1 -- 172.21.15.136:0/944506078 wait complete. 2024-03-25T18:39:14.002 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.000+0000 7f166a75d640 1 librados: shutdown 2024-03-25T18:39:14.011 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new d4d8f0df-36be-4807-b303-9edc2bb3b34e 4 2024-03-25T18:39:14.033 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:14.033+0000 7fe4ca82b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:39:14.067 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:14.067+0000 7f97b9810040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:39:14.112 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:14.112+0000 7f97b9810040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:39:14.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.151+0000 7f8f3653d640 10 monclient: get_monmap_and_config 2024-03-25T18:39:14.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.151+0000 7f8f3653d640 10 monclient: build_initial_monmap 2024-03-25T18:39:14.152 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.151+0000 7f8f3653d640 10 monclient: monmap: 2024-03-25T18:39:14.152 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:14.152 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.152 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:39:14.152523+0000 2024-03-25T18:39:14.152 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:39:14.152523+0000 2024-03-25T18:39:14.152 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:14.153 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.153 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:14.153 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:14.153 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:14.153 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.152+0000 7f8f3653d640 1 Processor -- start 2024-03-25T18:39:14.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.152+0000 7f8f3653d640 1 -- start start 2024-03-25T18:39:14.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.152+0000 7f8f3653d640 10 monclient: init 2024-03-25T18:39:14.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.152+0000 7f8f3653d640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:14.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:14.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f300b4700 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:14.154 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f30142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:14.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f300a96b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:14.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f300a9d60 con 0x7f8f300b4700 2024-03-25T18:39:14.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f3013ff10 con 0x7f8f30142a20 2024-03-25T18:39:14.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f300aa1f0 con 0x7f8f300a96b0 2024-03-25T18:39:14.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:14.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f3653d640 10 monclient(hunting): authenticate will time out at 6041.318872s 2024-03-25T18:39:14.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f2f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8f30142a20 0x7f8f3013f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38544/0 (socket says 172.21.15.136:38544) 2024-03-25T18:39:14.155 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.153+0000 7f8f2f7fe640 1 -- 172.21.15.136:0/909602950 learned_addr learned my addr 172.21.15.136:0/909602950 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.155+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 671262461 0 0) 0x7f8f3013ff10 con 0x7f8f30142a20 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f18003520 con 0x7f8f30142a20 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995008439 0 0) 0x7f8f300a9d60 con 0x7f8f300b4700 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.157 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:39:14.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f3013ff10 con 0x7f8f300b4700 2024-03-25T18:39:14.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424264350 0 0) 0x7f8f300aa1f0 con 0x7f8f300a96b0 2024-03-25T18:39:14.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:39:14.158 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.156+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f300a9d60 con 0x7f8f300a96b0 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 76334859 0 0) 0x7f8f3013ff10 con 0x7f8f300b4700 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 >> v1:172.21.15.32:6790/0 conn(0x7f8f300a96b0 legacy=0x7f8f3013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 >> v1:172.21.15.136:6789/0 conn(0x7f8f30142a20 legacy=0x7f8f3013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 10 monclient: _finish_hunting 0 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 1 monclient: found mon.noname-a 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f300b4ae0 con 0x7f8f300b4700 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 10 monclient: _finish_auth 0 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 10 monclient: _check_auth_tickets 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:14.160 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8f20001670 con 0x7f8f300b4700 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f2effd640 10 monclient: dump: 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.161 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f3653d640 5 monclient: authenticate success, global_id 4140 2024-03-25T18:39:14.162 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.159+0000 7f8f3653d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f200017d0 con 0x7f8f300b4700 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f2effd640 1 -- 172.21.15.136:0/909602950 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8f20001be0 con 0x7f8f300b4700 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 4 set_mon_vals no callback set 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f2effd640 10 monclient: dump: 2024-03-25T18:39:14.165 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 10 monclient: shutdown 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 1 -- 172.21.15.136:0/909602950 >> v1:172.21.15.32:6789/0 conn(0x7f8f300b4700 legacy=0x7f8f30142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.166 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 1 -- 172.21.15.136:0/909602950 shutdown_connections 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 1 -- 172.21.15.136:0/909602950 >> 172.21.15.136:0/909602950 conn(0x7f8f300668e0 msgr2=0x7f8f30066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 1 -- 172.21.15.136:0/909602950 shutdown_connections 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.161+0000 7f8f3653d640 1 -- 172.21.15.136:0/909602950 wait complete. 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.162+0000 7f8f3653d640 10 monclient: build_initial_monmap 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.162+0000 7f8f3653d640 10 monclient: monmap: 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:14.167 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.162+0000 7f8f3653d640 1 librados: starting msgr at 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.162+0000 7f8f3653d640 1 librados: starting objecter 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.162+0000 7f8f3653d640 1 Processor -- start 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 1 -- start start 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 1 librados: setting wanted keys 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 1 librados: calling monclient init 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 10 monclient: init 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f300a96b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f300b4700 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f30142a20 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f301d7660 con 0x7f8f30142a20 2024-03-25T18:39:14.168 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f301d8200 con 0x7f8f300b4700 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f301d72f0 con 0x7f8f300a96b0 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.163+0000 7f8f3653d640 10 monclient(hunting): authenticate will time out at 6041.329379s 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.164+0000 7f8f2f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8f300b4700 0x7f8f300abaf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38546/0 (socket says 172.21.15.136:38546) 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.164+0000 7f8f2f7fe640 1 -- 172.21.15.136:0/2800006861 learned_addr learned my addr 172.21.15.136:0/2800006861 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.164+0000 7f8f2cff9640 10 client.?.objecter ms_handle_connect 0x7f8f300b4700 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.164+0000 7f8f2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.164+0000 7f8f2cff9640 10 client.?.objecter ms_handle_connect 0x7f8f30142a20 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.164+0000 7f8f2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.164+0000 7f8f2cff9640 10 client.?.objecter ms_handle_connect 0x7f8f300a96b0 2024-03-25T18:39:14.169 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.164+0000 7f8f2cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1996005337 0 0) 0x7f8f301d7660 con 0x7f8f30142a20 2024-03-25T18:39:14.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.170 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f1c004eb0 con 0x7f8f30142a20 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085059545 0 0) 0x7f8f301d8200 con 0x7f8f300b4700 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f301d7660 con 0x7f8f300b4700 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1512025188 0 0) 0x7f8f301d72f0 con 0x7f8f300a96b0 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:39:14.171 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.166+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f301d8200 con 0x7f8f300a96b0 2024-03-25T18:39:14.172 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.170+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4204884425 0 0) 0x7f8f301d7660 con 0x7f8f300b4700 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.170+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f301d72f0 con 0x7f8f300b4700 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.170+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8f240018d0 con 0x7f8f300b4700 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.170+0000 7f8f2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.170+0000 7f8f2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.170+0000 7f8f2cff9640 10 monclient(hunting): dump: 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.173 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.171+0000 7f8f2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.171+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2297776681 0 0) 0x7f8f1c004eb0 con 0x7f8f30142a20 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.171+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f301d7660 con 0x7f8f30142a20 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.171+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8f28001bd0 con 0x7f8f30142a20 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.171+0000 7f8f2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.171+0000 7f8f2cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.171+0000 7f8f2cff9640 10 monclient(hunting): dump: 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.174 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.172+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120773129 0 0) 0x7f8f301d8200 con 0x7f8f300a96b0 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.172+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f1c004eb0 con 0x7f8f300a96b0 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.172+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8f20001ae0 con 0x7f8f300a96b0 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.172+0000 7f8f2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.172+0000 7f8f2cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.172+0000 7f8f2cff9640 10 monclient(hunting): dump: 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.175 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.176 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.176 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.176 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.176 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2670803928 0 0) 0x7f8f301d7660 con 0x7f8f30142a20 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 >> v1:172.21.15.32:6790/0 conn(0x7f8f300a96b0 legacy=0x7f8f301d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 >> v1:172.21.15.136:6789/0 conn(0x7f8f300b4700 legacy=0x7f8f300abaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 1 monclient: found mon.a 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f300ac200 con 0x7f8f30142a20 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 10 monclient: _finish_auth 0 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:39:14.177 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 5 monclient: authenticate success, global_id 4143 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 10 monclient: _renew_subs 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f30136800 con 0x7f8f30142a20 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 10 client.4143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 10 monclient: _renew_subs 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f30133fb0 con 0x7f8f30142a20 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.174+0000 7f8f3653d640 1 librados: init done 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.175+0000 7f8f3653d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.175+0000 7f8f3653d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.175+0000 7f8f3653d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:14.178 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.175+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f30134200 con 0x7f8f30142a20 2024-03-25T18:39:14.210 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:14.210+0000 7fe4ca82b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:39:14.254 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:14.254+0000 7fe4ca82b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:39:14.270 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:14.269+0000 7f97b9810040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:39:14.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.277+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f28002060 con 0x7f8f30142a20 2024-03-25T18:39:14.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.277+0000 7f8f2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:14.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.277+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8f28003b30 con 0x7f8f30142a20 2024-03-25T18:39:14.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.277+0000 7f8f2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.277+0000 7f8f2cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:14.278 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.277+0000 7f8f2cff9640 10 monclient: dump: 2024-03-25T18:39:14.278 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.278 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.277+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8f28004020 con 0x7f8f30142a20 2024-03-25T18:39:14.279 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.277+0000 7f8f2e7fc640 4 set_mon_vals no callback set 2024-03-25T18:39:14.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.278+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2864891154 0 0) 0x7f8f280056e0 con 0x7f8f30142a20 2024-03-25T18:39:14.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.278+0000 7f8f2cff9640 10 client.4143.objecter ms_dispatch 0x7f8f30000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:39:14.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.278+0000 7f8f2cff9640 3 client.4143.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:39:14.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.278+0000 7f8f2cff9640 3 client.4143.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:39:14.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.278+0000 7f8f2cff9640 20 client.4143.objecter dump_active .. 0 homeless 2024-03-25T18:39:14.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.278+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8f280051f0 con 0x7f8f30142a20 2024-03-25T18:39:14.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.278+0000 7f8f2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:14.280 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.278+0000 7f8f2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:14.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.349+0000 7f8f3653d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d4d8f0df-36be-4807-b303-9edc2bb3b34e", "id": 4}] 2024-03-25T18:39:14.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.349+0000 7f8f3653d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d4d8f0df-36be-4807-b303-9edc2bb3b34e", "id": 4}] 2024-03-25T18:39:14.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.349+0000 7f8f3653d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:14.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.349+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d4d8f0df-36be-4807-b303-9edc2bb3b34e", "id": 4} v 0) v1 -- 0x7f8f301d7480 con 0x7f8f30142a20 2024-03-25T18:39:14.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.385+0000 7f8f2cff9640 1 -- 172.21.15.136:0/2800006861 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d4d8f0df-36be-4807-b303-9edc2bb3b34e", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2751195039 0 1697012583) 0x7f8f280059d0 con 0x7f8f30142a20 2024-03-25T18:39:14.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.385+0000 7f8f2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d4d8f0df-36be-4807-b303-9edc2bb3b34e", "id": 4}] 2024-03-25T18:39:14.386 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.385+0000 7f8f2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:14.387 INFO:teuthology.orchestra.run.smithi136.stdout:4 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 10 librados: watch_flush enter 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 10 librados: watch_flush exit 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 20 client.4143.objecter shutdown clearing up homeless session... 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 10 client.4143.objecter successfully canceled tick 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 10 monclient: shutdown 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 >> v1:172.21.15.32:6789/0 conn(0x7f8f30142a20 legacy=0x7f8f301e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 shutdown_connections 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.386+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 >> 172.21.15.136:0/2800006861 conn(0x7f8f300668e0 msgr2=0x7f8f300b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.387+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 shutdown_connections 2024-03-25T18:39:14.388 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.387+0000 7f8f3653d640 1 -- 172.21.15.136:0/2800006861 wait complete. 2024-03-25T18:39:14.389 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.387+0000 7f8f3653d640 1 librados: shutdown 2024-03-25T18:39:14.398 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new c05dfe41-a0ff-4367-a4e9-0173990d2eaf 5 2024-03-25T18:39:14.434 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:14.434+0000 7fe4ca82b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.536+0000 7f29690fb640 10 monclient: get_monmap_and_config 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.536+0000 7f29690fb640 10 monclient: build_initial_monmap 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.536+0000 7f29690fb640 10 monclient: monmap: 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:39:14.536971+0000 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:39:14.536971+0000 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:14.537 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.538 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:14.538 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:14.538 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:14.538 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.536+0000 7f29690fb640 1 Processor -- start 2024-03-25T18:39:14.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.536+0000 7f29690fb640 1 -- start start 2024-03-25T18:39:14.538 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.536+0000 7f29690fb640 10 monclient: init 2024-03-25T18:39:14.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:14.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:39:14.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 10 monclient(hunting): picked mon.noname-b con 0x7f29640b4700 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:14.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 10 monclient(hunting): picked mon.noname-c con 0x7f2964142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:14.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 10 monclient(hunting): picked mon.noname-a con 0x7f29640a96b0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:14.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29640a9d60 con 0x7f29640a96b0 2024-03-25T18:39:14.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f296413ff10 con 0x7f2964142a20 2024-03-25T18:39:14.539 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29640aa1f0 con 0x7f29640b4700 2024-03-25T18:39:14.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:14.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f29690fb640 10 monclient(hunting): authenticate will time out at 6041.703128s 2024-03-25T18:39:14.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f296259c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2964142a20 0x7f296413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38550/0 (socket says 172.21.15.136:38550) 2024-03-25T18:39:14.540 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.537+0000 7f296259c640 1 -- 172.21.15.136:0/1571815146 learned_addr learned my addr 172.21.15.136:0/1571815146 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591876977 0 0) 0x7f29640a9d60 con 0x7f29640a96b0 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f294c003520 con 0x7f29640a96b0 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678386493 0 0) 0x7f296413ff10 con 0x7f2964142a20 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.541 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:39:14.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29640a9d60 con 0x7f2964142a20 2024-03-25T18:39:14.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2954215171 0 0) 0x7f29640aa1f0 con 0x7f29640b4700 2024-03-25T18:39:14.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:39:14.542 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.540+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296413ff10 con 0x7f29640b4700 2024-03-25T18:39:14.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3538831959 0 0) 0x7f29640a9d60 con 0x7f2964142a20 2024-03-25T18:39:14.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 >> v1:172.21.15.32:6790/0 conn(0x7f29640b4700 legacy=0x7f2964142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 >> v1:172.21.15.32:6789/0 conn(0x7f29640a96b0 legacy=0x7f296413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:39:14.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 1 monclient: found mon.noname-c 2024-03-25T18:39:14.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:14.544 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29640b4ae0 con 0x7f2964142a20 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 10 monclient: _finish_auth 0 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2958001670 con 0x7f2964142a20 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f2961d9b640 10 monclient: dump: 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.545 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.546 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.546 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.546 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.546 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f29690fb640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:39:14.546 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.543+0000 7f29690fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.545+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29580017d0 con 0x7f2964142a20 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.545+0000 7f2961d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.545+0000 7f2961d9b640 1 -- 172.21.15.136:0/1571815146 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2958001be0 con 0x7f2964142a20 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.545+0000 7f2961d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.545+0000 7f2961d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.545+0000 7f2961d9b640 10 monclient: dump: 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.548 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.545+0000 7f29690fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.545+0000 7f29690fb640 4 set_mon_vals no callback set 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.546+0000 7f29690fb640 10 monclient: shutdown 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.546+0000 7f29690fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.546+0000 7f29690fb640 1 -- 172.21.15.136:0/1571815146 >> v1:172.21.15.136:6789/0 conn(0x7f2964142a20 legacy=0x7f296413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.546+0000 7f29690fb640 1 -- 172.21.15.136:0/1571815146 shutdown_connections 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.546+0000 7f29690fb640 1 -- 172.21.15.136:0/1571815146 >> 172.21.15.136:0/1571815146 conn(0x7f29640668e0 msgr2=0x7f2964066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.546+0000 7f29690fb640 1 -- 172.21.15.136:0/1571815146 shutdown_connections 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.546+0000 7f29690fb640 1 -- 172.21.15.136:0/1571815146 wait complete. 2024-03-25T18:39:14.549 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 10 monclient: build_initial_monmap 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 10 monclient: monmap: 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 1 librados: starting msgr at 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 1 librados: starting objecter 2024-03-25T18:39:14.550 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 1 Processor -- start 2024-03-25T18:39:14.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 1 -- start start 2024-03-25T18:39:14.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 1 librados: setting wanted keys 2024-03-25T18:39:14.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 1 librados: calling monclient init 2024-03-25T18:39:14.551 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.547+0000 7f29690fb640 10 monclient: init 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 10 monclient(hunting): picked mon.noname-a con 0x7f29640a96b0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 10 monclient(hunting): picked mon.noname-c con 0x7f29640b4700 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 10 monclient(hunting): picked mon.noname-b con 0x7f2964142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29641daba0 con 0x7f29640a96b0 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29640acf70 con 0x7f2964142a20 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29641db110 con 0x7f29640b4700 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:14.553 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f29690fb640 10 monclient(hunting): authenticate will time out at 6041.714143s 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f296359e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2964142a20 0x7f29640ac860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38562/0 (socket says 172.21.15.136:38562) 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.548+0000 7f296359e640 1 -- 172.21.15.136:0/2295140060 learned_addr learned my addr 172.21.15.136:0/2295140060 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.549+0000 7f294b7fe640 10 client.?.objecter ms_handle_connect 0x7f2964142a20 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.549+0000 7f294b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.549+0000 7f294b7fe640 10 client.?.objecter ms_handle_connect 0x7f29640a96b0 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.549+0000 7f294b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.549+0000 7f294b7fe640 10 client.?.objecter ms_handle_connect 0x7f29640b4700 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.549+0000 7f294b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039741998 0 0) 0x7f29641daba0 con 0x7f29640a96b0 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.554 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2950004eb0 con 0x7f29640a96b0 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048678440 0 0) 0x7f29640acf70 con 0x7f2964142a20 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29641daba0 con 0x7f2964142a20 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2552940154 0 0) 0x7f29641db110 con 0x7f29640b4700 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:39:14.555 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.551+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29640acf70 con 0x7f29640b4700 2024-03-25T18:39:14.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.555+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1859001343 0 0) 0x7f2950004eb0 con 0x7f29640a96b0 2024-03-25T18:39:14.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.555+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29641db110 con 0x7f29640a96b0 2024-03-25T18:39:14.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.555+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f29540018f0 con 0x7f29640a96b0 2024-03-25T18:39:14.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.555+0000 7f294b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.555+0000 7f294b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:14.556 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.555+0000 7f294b7fe640 10 monclient(hunting): dump: 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.555+0000 7f294b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.556+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1478238780 0 0) 0x7f29641daba0 con 0x7f2964142a20 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.556+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2950004eb0 con 0x7f2964142a20 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.556+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f295c001bf0 con 0x7f2964142a20 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.556+0000 7f294b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.556+0000 7f294b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:14.557 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.556+0000 7f294b7fe640 10 monclient(hunting): dump: 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.558 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.558+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3655874110 0 0) 0x7f29640acf70 con 0x7f29640b4700 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.558+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29641daba0 con 0x7f29640b4700 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.558+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2958001ae0 con 0x7f29640b4700 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.558+0000 7f294b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.558+0000 7f294b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.558+0000 7f294b7fe640 10 monclient(hunting): dump: 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.560 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1661507059 0 0) 0x7f2950004eb0 con 0x7f2964142a20 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 >> v1:172.21.15.32:6790/0 conn(0x7f29640b4700 legacy=0x7f296413a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 >> v1:172.21.15.32:6789/0 conn(0x7f29640a96b0 legacy=0x7f29640abd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 1 monclient: found mon.b 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29641db2a0 con 0x7f2964142a20 2024-03-25T18:39:14.561 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f294b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 10 monclient: _renew_subs 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29641db4f0 con 0x7f2964142a20 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 10 monclient: _renew_subs 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29641362d0 con 0x7f2964142a20 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 1 librados: init done 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:14.562 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.563 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.559+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29640b1dc0 con 0x7f2964142a20 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.656+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f295c002060 con 0x7f2964142a20 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.656+0000 7f294b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.656+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f295c003b30 con 0x7f2964142a20 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.656+0000 7f294b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.656+0000 7f294b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.656+0000 7f294b7fe640 10 monclient: dump: 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.658 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f295c0040e0 con 0x7f2964142a20 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 3162200744 0 0) 0x7f295c006940 con 0x7f2964142a20 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 10 client.4141.objecter ms_dispatch 0x7f2964000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.656+0000 7f296159a640 4 set_mon_vals no callback set 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f295c018de0 con 0x7f2964142a20 2024-03-25T18:39:14.659 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:14.660 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.657+0000 7f294b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:14.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.729+0000 7f29690fb640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "c05dfe41-a0ff-4367-a4e9-0173990d2eaf", "id": 5}] 2024-03-25T18:39:14.729 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.729+0000 7f29690fb640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "c05dfe41-a0ff-4367-a4e9-0173990d2eaf", "id": 5}] 2024-03-25T18:39:14.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.729+0000 7f29690fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.730 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.729+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "c05dfe41-a0ff-4367-a4e9-0173990d2eaf", "id": 5} v 0) v1 -- 0x7f29641dad90 con 0x7f2964142a20 2024-03-25T18:39:14.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.821+0000 7f294b7fe640 1 -- 172.21.15.136:0/2295140060 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "c05dfe41-a0ff-4367-a4e9-0173990d2eaf", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 2806166958 0 1988412176) 0x7f295c004690 con 0x7f2964142a20 2024-03-25T18:39:14.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.821+0000 7f294b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "c05dfe41-a0ff-4367-a4e9-0173990d2eaf", "id": 5}] 2024-03-25T18:39:14.822 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.821+0000 7f294b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:14.823 INFO:teuthology.orchestra.run.smithi136.stdout:5 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 10 librados: watch_flush enter 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 10 librados: watch_flush exit 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 10 client.4141.objecter successfully canceled tick 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 10 monclient: shutdown 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 >> v1:172.21.15.136:6789/0 conn(0x7f2964142a20 legacy=0x7f29640ac860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 shutdown_connections 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 >> 172.21.15.136:0/2295140060 conn(0x7f29640668e0 msgr2=0x7f296413f7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 shutdown_connections 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 1 -- 172.21.15.136:0/2295140060 wait complete. 2024-03-25T18:39:14.824 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.822+0000 7f29690fb640 1 librados: shutdown 2024-03-25T18:39:14.833 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new ab217fa6-09ba-4227-87a1-9e5d660603d1 6 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.934+0000 7f00c456b640 10 monclient: get_monmap_and_config 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.934+0000 7f00c456b640 10 monclient: build_initial_monmap 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.934+0000 7f00c456b640 10 monclient: monmap: 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:39:14.935023+0000 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:39:14.935023+0000 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:14.935 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.936 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:14.936 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:14.936 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:14.936 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.934+0000 7f00c456b640 1 Processor -- start 2024-03-25T18:39:14.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.934+0000 7f00c456b640 1 -- start start 2024-03-25T18:39:14.936 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.934+0000 7f00c456b640 10 monclient: init 2024-03-25T18:39:14.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:14.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:39:14.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 10 monclient(hunting): picked mon.noname-c con 0x7f00bc06c2b0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:14.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 10 monclient(hunting): picked mon.noname-a con 0x7f00bc0b0100 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:14.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 10 monclient(hunting): picked mon.noname-b con 0x7f00bc146300 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:14.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc13feb0 con 0x7f00bc0b0100 2024-03-25T18:39:14.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc06d050 con 0x7f00bc06c2b0 2024-03-25T18:39:14.937 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc0b06d0 con 0x7f00bc146300 2024-03-25T18:39:14.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:14.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.935+0000 7f00c456b640 10 monclient(hunting): authenticate will time out at 6042.101522s 2024-03-25T18:39:14.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.936+0000 7f00c1b06640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f00bc0b0100 0x7f00bc145bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.136:52556/0 (socket says 172.21.15.136:52556) 2024-03-25T18:39:14.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.936+0000 7f00c1b06640 1 -- 172.21.15.136:0/3250476836 learned_addr learned my addr 172.21.15.136:0/3250476836 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:14.938 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.936+0000 7f00c2b08640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f00bc146300 0x7f00bc14db40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.136:35032/0 (socket says 172.21.15.136:35032) 2024-03-25T18:39:14.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.938+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1089427471 0 0) 0x7f00bc06d050 con 0x7f00bc06c2b0 2024-03-25T18:39:14.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:39:14.939 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00a8003520 con 0x7f00bc06c2b0 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2699762910 0 0) 0x7f00bc0b06d0 con 0x7f00bc146300 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00bc06d050 con 0x7f00bc146300 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3270223962 0 0) 0x7f00bc13feb0 con 0x7f00bc0b0100 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.940 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:39:14.941 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.939+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00bc0b06d0 con 0x7f00bc0b0100 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3462647852 0 0) 0x7f00a8003520 con 0x7f00bc06c2b0 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 >> v1:172.21.15.32:6790/0 conn(0x7f00bc146300 legacy=0x7f00bc14db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 >> v1:172.21.15.32:6789/0 conn(0x7f00bc0b0100 legacy=0x7f00bc145bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 10 monclient: _finish_hunting 0 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 1 monclient: found mon.noname-c 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc06d4a0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 10 monclient: _finish_auth 0 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 10 monclient: _check_auth_tickets 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:14.943 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f00ac001690 con 0x7f00bc06c2b0 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c1305640 10 monclient: dump: 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c456b640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:39:14.944 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.942+0000 7f00c456b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00ac0017f0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c1305640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c1305640 1 -- 172.21.15.136:0/3250476836 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f00ac001c00 con 0x7f00bc06c2b0 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c1305640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c1305640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c1305640 10 monclient: dump: 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.946 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c456b640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c456b640 4 set_mon_vals no callback set 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c456b640 10 monclient: shutdown 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c456b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.944+0000 7f00c456b640 1 -- 172.21.15.136:0/3250476836 >> v1:172.21.15.136:6789/0 conn(0x7f00bc06c2b0 legacy=0x7f00bc0af9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.945+0000 7f00c456b640 1 -- 172.21.15.136:0/3250476836 shutdown_connections 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.945+0000 7f00c456b640 1 -- 172.21.15.136:0/3250476836 >> 172.21.15.136:0/3250476836 conn(0x7f00bc0b5cd0 msgr2=0x7f00bc0b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.945+0000 7f00c456b640 1 -- 172.21.15.136:0/3250476836 shutdown_connections 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.945+0000 7f00c456b640 1 -- 172.21.15.136:0/3250476836 wait complete. 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.945+0000 7f00c456b640 10 monclient: build_initial_monmap 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.945+0000 7f00c456b640 10 monclient: monmap: 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:14.947 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:14.948 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-25T18:39:14.948 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-25T18:39:14.948 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:14.948 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-25T18:39:14.948 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:14.948 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:14.948 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:14.948 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 librados: starting msgr at 2024-03-25T18:39:14.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 librados: starting objecter 2024-03-25T18:39:14.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 Processor -- start 2024-03-25T18:39:14.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 -- start start 2024-03-25T18:39:14.950 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 librados: setting wanted keys 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 librados: calling monclient init 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 10 monclient: init 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 10 monclient(hunting): picked mon.noname-b con 0x7f00bc06c2b0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 10 monclient(hunting): picked mon.noname-c con 0x7f00bc0b0100 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 10 monclient(hunting): picked mon.noname-a con 0x7f00bc146300 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc11e260 con 0x7f00bc146300 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc1d7db0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00bc13b910 con 0x7f00bc0b0100 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.946+0000 7f00c456b640 10 monclient(hunting): authenticate will time out at 6042.112542s 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.947+0000 7f00c2307640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f00bc06c2b0 0x7f00bc1d75f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38586/0 (socket says 172.21.15.136:38586) 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.947+0000 7f00c2307640 1 -- 172.21.15.136:0/1329705453 learned_addr learned my addr 172.21.15.136:0/1329705453 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.947+0000 7f00b2ffd640 10 client.?.objecter ms_handle_connect 0x7f00bc146300 2024-03-25T18:39:14.951 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.947+0000 7f00b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.947+0000 7f00b2ffd640 10 client.?.objecter ms_handle_connect 0x7f00bc0b0100 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.947+0000 7f00b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.947+0000 7f00b2ffd640 10 client.?.objecter ms_handle_connect 0x7f00bc06c2b0 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.947+0000 7f00b2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 365050328 0 0) 0x7f00bc1d7db0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00a0004eb0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939668039 0 0) 0x7f00bc11e260 con 0x7f00bc146300 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.949+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00bc1d7db0 con 0x7f00bc146300 2024-03-25T18:39:14.952 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.950+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3388632075 0 0) 0x7f00bc13b910 con 0x7f00bc0b0100 2024-03-25T18:39:14.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.950+0000 7f00b2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:14.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.950+0000 7f00b2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:14.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.950+0000 7f00b2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:14.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.950+0000 7f00b2ffd640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:39:14.953 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.950+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00bc11e260 con 0x7f00bc0b0100 2024-03-25T18:39:14.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.953+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2217956639 0 0) 0x7f00a0004eb0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.953+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00bc13b910 con 0x7f00bc06c2b0 2024-03-25T18:39:14.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.954+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f00ac001c00 con 0x7f00bc06c2b0 2024-03-25T18:39:14.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.954+0000 7f00b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.954 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.954+0000 7f00b2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.954+0000 7f00b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.955 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.954+0000 7f00b2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.956+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3962108299 0 0) 0x7f00bc1d7db0 con 0x7f00bc146300 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.956+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00a0004eb0 con 0x7f00bc146300 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.956+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1742270432 0 0) 0x7f00bc11e260 con 0x7f00bc0b0100 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.956+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f00bc1d7db0 con 0x7f00bc0b0100 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.956+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f00b80017f0 con 0x7f00bc0b0100 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.956+0000 7f00b2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.956+0000 7f00b2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.956+0000 7f00b2ffd640 10 monclient(hunting): dump: 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.957 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.958 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.958 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.958 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.958 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.958 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.958 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.958 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1342809687 0 0) 0x7f00bc13b910 con 0x7f00bc06c2b0 2024-03-25T18:39:14.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 >> v1:172.21.15.32:6790/0 conn(0x7f00bc0b0100 legacy=0x7f00bc13ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.959 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 >> v1:172.21.15.32:6789/0 conn(0x7f00bc146300 legacy=0x7f00bc1400d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 1 monclient: found mon.b 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00bc13baa0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00b2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 10 monclient: _renew_subs 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00bc13b490 con 0x7f00bc06c2b0 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 10 monclient: _renew_subs 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00bc070ec0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.960 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.957+0000 7f00c456b640 1 librados: init done 2024-03-25T18:39:14.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.958+0000 7f00c456b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:14.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.958+0000 7f00c456b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:14.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.958+0000 7f00c456b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:14.961 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.958+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00bc1d7ff0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f00ac0038a0 con 0x7f00bc06c2b0 2024-03-25T18:39:14.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:14.962 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f00ac003a90 con 0x7f00bc06c2b0 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 10 monclient: dump: 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:14.963 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00c0b04640 4 set_mon_vals no callback set 2024-03-25T18:39:14.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f00ac004040 con 0x7f00bc06c2b0 2024-03-25T18:39:14.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 4285876336 0 0) 0x7f00ac006200 con 0x7f00bc06c2b0 2024-03-25T18:39:14.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 10 client.4147.objecter ms_dispatch 0x7f00bc000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:39:14.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:39:14.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:39:14.964 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:14.961+0000 7f00b2ffd640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T18:39:15.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.054+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f00bc13baa0 con 0x7f00bc06c2b0 2024-03-25T18:39:15.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.054+0000 7f00b2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:15.054 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.054+0000 7f00b2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:15.114 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:15.114+0000 7f97b9810040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:39:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.132+0000 7f00c456b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ab217fa6-09ba-4227-87a1-9e5d660603d1", "id": 6}] 2024-03-25T18:39:15.132 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.132+0000 7f00c456b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ab217fa6-09ba-4227-87a1-9e5d660603d1", "id": 6}] 2024-03-25T18:39:15.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.132+0000 7f00c456b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:15.133 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.132+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ab217fa6-09ba-4227-87a1-9e5d660603d1", "id": 6} v 0) v1 -- 0x7f00bc04f730 con 0x7f00bc06c2b0 2024-03-25T18:39:15.224 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.224+0000 7f00b2ffd640 1 -- 172.21.15.136:0/1329705453 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ab217fa6-09ba-4227-87a1-9e5d660603d1", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 1793136238 0 1113811849) 0x7f00ac0042f0 con 0x7f00bc06c2b0 2024-03-25T18:39:15.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.224+0000 7f00b2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ab217fa6-09ba-4227-87a1-9e5d660603d1", "id": 6}] 2024-03-25T18:39:15.225 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.224+0000 7f00b2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:15.225 INFO:teuthology.orchestra.run.smithi136.stdout:6 2024-03-25T18:39:15.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 10 librados: watch_flush enter 2024-03-25T18:39:15.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 10 librados: watch_flush exit 2024-03-25T18:39:15.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T18:39:15.226 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 10 client.4147.objecter successfully canceled tick 2024-03-25T18:39:15.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 10 monclient: shutdown 2024-03-25T18:39:15.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:15.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 >> v1:172.21.15.136:6789/0 conn(0x7f00bc06c2b0 legacy=0x7f00bc1d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 shutdown_connections 2024-03-25T18:39:15.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 >> 172.21.15.136:0/1329705453 conn(0x7f00bc0b5cd0 msgr2=0x7f00bc0b35d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:15.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 shutdown_connections 2024-03-25T18:39:15.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 1 -- 172.21.15.136:0/1329705453 wait complete. 2024-03-25T18:39:15.227 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.225+0000 7f00c456b640 1 librados: shutdown 2024-03-25T18:39:15.235 DEBUG:teuthology.orchestra.run.smithi136:> sudo ceph --cluster ceph osd new 6320cd08-17e0-4bc6-8b8e-9d8341a59cd1 7 2024-03-25T18:39:15.298 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:15.298+0000 7fe4ca82b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:39:15.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.333+0000 7fd6232a5640 10 monclient: get_monmap_and_config 2024-03-25T18:39:15.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.333+0000 7fd6232a5640 10 monclient: build_initial_monmap 2024-03-25T18:39:15.334 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.333+0000 7fd6232a5640 10 monclient: monmap: 2024-03-25T18:39:15.334 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:15.334 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.334 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:39:15.334570+0000 2024-03-25T18:39:15.335 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:39:15.334570+0000 2024-03-25T18:39:15.335 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:15.335 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:15.335 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:15.335 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:15.335 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:15.335 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.334+0000 7fd6232a5640 1 Processor -- start 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.334+0000 7fd6232a5640 1 -- start start 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.334+0000 7fd6232a5640 10 monclient: init 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 10 monclient(hunting): picked mon.noname-c con 0x7fd61c0a9920 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 10 monclient(hunting): picked mon.noname-a con 0x7fd61c0b45a0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 10 monclient(hunting): picked mon.noname-b con 0x7fd61c146450 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c06b100 con 0x7fd61c0b45a0 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c0b2940 con 0x7fd61c0a9920 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c13fec0 con 0x7fd61c146450 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd6232a5640 10 monclient(hunting): authenticate will time out at 6042.500965s 2024-03-25T18:39:15.337 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd621041640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd61c0a9920 0x7fd61c13f7b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38596/0 (socket says 172.21.15.136:38596) 2024-03-25T18:39:15.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.335+0000 7fd621041640 1 -- 172.21.15.136:0/3706789470 learned_addr learned my addr 172.21.15.136:0/3706789470 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:15.338 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097305056 0 0) 0x7fd61c06b100 con 0x7fd61c0b45a0 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd608003520 con 0x7fd61c0b45a0 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247009572 0 0) 0x7fd61c0b2940 con 0x7fd61c0a9920 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:39:15.339 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.338+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd61c06b100 con 0x7fd61c0a9920 2024-03-25T18:39:15.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.339+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051172503 0 0) 0x7fd61c13fec0 con 0x7fd61c146450 2024-03-25T18:39:15.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.339+0000 7fd613fff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:15.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.339+0000 7fd613fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:15.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.339+0000 7fd613fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:15.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.339+0000 7fd613fff640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:39:15.340 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.339+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd61c0b2940 con 0x7fd61c146450 2024-03-25T18:39:15.342 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 248004856 0 0) 0x7fd608003520 con 0x7fd61c0b45a0 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 >> v1:172.21.15.32:6790/0 conn(0x7fd61c146450 legacy=0x7fd61c14dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 >> v1:172.21.15.136:6789/0 conn(0x7fd61c0a9920 legacy=0x7fd61c13f7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 10 monclient: _finish_hunting 0 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 1 monclient: found mon.noname-a 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c140050 con 0x7fd61c0b45a0 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 10 monclient: _finish_auth 0 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 10 monclient: _check_auth_tickets 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd618001690 con 0x7fd61c0b45a0 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:15.343 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd613fff640 10 monclient: dump: 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd6232a5640 5 monclient: authenticate success, global_id 4158 2024-03-25T18:39:15.344 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.341+0000 7fd6232a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.343+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd6180017f0 con 0x7fd61c0b45a0 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.343+0000 7fd613fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.343+0000 7fd613fff640 1 -- 172.21.15.136:0/3706789470 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd618001a60 con 0x7fd61c0b45a0 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.343+0000 7fd613fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.343+0000 7fd613fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.343+0000 7fd613fff640 10 monclient: dump: 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.348 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.343+0000 7fd6232a5640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.343+0000 7fd6232a5640 4 set_mon_vals no callback set 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.344+0000 7fd6232a5640 10 monclient: shutdown 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.344+0000 7fd6232a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.344+0000 7fd6232a5640 1 -- 172.21.15.136:0/3706789470 >> v1:172.21.15.32:6789/0 conn(0x7fd61c0b45a0 legacy=0x7fd61c145d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.344+0000 7fd6232a5640 1 -- 172.21.15.136:0/3706789470 shutdown_connections 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.344+0000 7fd6232a5640 1 -- 172.21.15.136:0/3706789470 >> 172.21.15.136:0/3706789470 conn(0x7fd61c0b7f40 msgr2=0x7fd61c0b8350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:15.349 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.344+0000 7fd6232a5640 1 -- 172.21.15.136:0/3706789470 shutdown_connections 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.344+0000 7fd6232a5640 1 -- 172.21.15.136:0/3706789470 wait complete. 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 10 monclient: build_initial_monmap 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 10 monclient: monmap: 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 0 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 0.000000 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:created 0.000000 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 1 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:15.350 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 1 librados: starting msgr at 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 1 librados: starting objecter 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 1 Processor -- start 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 1 -- start start 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 1 librados: setting wanted keys 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 1 librados: calling monclient init 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.345+0000 7fd6232a5640 10 monclient: init 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 10 monclient(hunting): picked mon.noname-c con 0x7fd61c0a9920 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 10 monclient(hunting): picked mon.noname-a con 0x7fd61c0b45a0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 10 monclient(hunting): picked mon.noname-b con 0x7fd61c146450 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c135a90 con 0x7fd61c0b45a0 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c142c40 con 0x7fd61c146450 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd61c142720 con 0x7fd61c0a9920 2024-03-25T18:39:15.351 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd6232a5640 10 monclient(hunting): authenticate will time out at 6042.512083s 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd621842640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd61c146450 0x7fd61c135380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.136:38612/0 (socket says 172.21.15.136:38612) 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.346+0000 7fd621842640 1 -- 172.21.15.136:0/2788248085 learned_addr learned my addr 172.21.15.136:0/2788248085 (peer_addr_for_me v1:172.21.15.136:0/0) 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.347+0000 7fd611ffb640 10 client.?.objecter ms_handle_connect 0x7fd61c0b45a0 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.347+0000 7fd611ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.347+0000 7fd611ffb640 10 client.?.objecter ms_handle_connect 0x7fd61c0a9920 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.347+0000 7fd611ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.347+0000 7fd611ffb640 10 client.?.objecter ms_handle_connect 0x7fd61c146450 2024-03-25T18:39:15.352 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.347+0000 7fd611ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:15.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.348+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417401043 0 0) 0x7fd61c142c40 con 0x7fd61c146450 2024-03-25T18:39:15.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.348+0000 7fd611ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:15.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.348+0000 7fd611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:15.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.348+0000 7fd611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:15.353 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.348+0000 7fd611ffb640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.348+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd600004eb0 con 0x7fd61c146450 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.349+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161609883 0 0) 0x7fd61c135a90 con 0x7fd61c0b45a0 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.349+0000 7fd611ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.349+0000 7fd611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.349+0000 7fd611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.349+0000 7fd611ffb640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.349+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd61c142c40 con 0x7fd61c0b45a0 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.350+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181662010 0 0) 0x7fd61c142720 con 0x7fd61c0a9920 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.350+0000 7fd611ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.350+0000 7fd611ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.350+0000 7fd611ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.350+0000 7fd611ffb640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:39:15.354 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.350+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd61c135a90 con 0x7fd61c0a9920 2024-03-25T18:39:15.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.353+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1693929311 0 0) 0x7fd600004eb0 con 0x7fd61c146450 2024-03-25T18:39:15.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.353+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd61c142720 con 0x7fd61c146450 2024-03-25T18:39:15.355 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.353+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd614001c70 con 0x7fd61c146450 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.353+0000 7fd611ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.353+0000 7fd611ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.353+0000 7fd611ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:15.356 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.353+0000 7fd611ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.354+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3671702710 0 0) 0x7fd61c142c40 con 0x7fd61c0b45a0 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.354+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd600004eb0 con 0x7fd61c0b45a0 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.354+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd618001960 con 0x7fd61c0b45a0 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.354+0000 7fd611ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.354+0000 7fd611ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.354+0000 7fd611ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.357 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.358 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:15.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 7060668 0 0) 0x7fd61c142720 con 0x7fd61c146450 2024-03-25T18:39:15.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 >> v1:172.21.15.32:6790/0 conn(0x7fd61c0a9920 legacy=0x7fd61c1d7210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 >> v1:172.21.15.32:6789/0 conn(0x7fd61c0b45a0 legacy=0x7fd61c134c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:39:15.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 1 monclient: found mon.b 2024-03-25T18:39:15.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:15.359 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd61c13be70 con 0x7fd61c146450 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 10 monclient: _finish_auth 0 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd611ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 10 monclient: _renew_subs 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd61c1d7920 con 0x7fd61c146450 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 10 monclient: _renew_subs 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd61c141c10 con 0x7fd61c146450 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.356+0000 7fd6232a5640 1 librados: init done 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.357+0000 7fd6232a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.357+0000 7fd6232a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:15.360 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.357+0000 7fd6232a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:15.361 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.357+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd61c143f50 con 0x7fd61c146450 2024-03-25T18:39:15.400 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:15.400+0000 7f97b9810040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:39:15.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.451+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd614003990 con 0x7fd61c146450 2024-03-25T18:39:15.452 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.451+0000 7fd611ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:15.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.451+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd614003d20 con 0x7fd61c146450 2024-03-25T18:39:15.453 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.451+0000 7fd611ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.453 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:15.452+0000 7f97b9810040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 10 monclient: dump: 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:epoch 1 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:election_strategy: 3 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.454 INFO:teuthology.orchestra.run.smithi136.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd6140042d0 con 0x7fd61c146450 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd6137fe640 4 set_mon_vals no callback set 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 424471745 0 0) 0x7fd614006b10 con 0x7fd61c146450 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 10 client.4153.objecter ms_dispatch 0x7fd61c000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd614018e00 con 0x7fd61c146450 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:15.455 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.452+0000 7fd611ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:15.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.523+0000 7fd6232a5640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "6320cd08-17e0-4bc6-8b8e-9d8341a59cd1", "id": 7}] 2024-03-25T18:39:15.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.523+0000 7fd6232a5640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "6320cd08-17e0-4bc6-8b8e-9d8341a59cd1", "id": 7}] 2024-03-25T18:39:15.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.523+0000 7fd6232a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:15.524 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.523+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd new", "uuid": "6320cd08-17e0-4bc6-8b8e-9d8341a59cd1", "id": 7} v 0) v1 -- 0x7fd61c13c060 con 0x7fd61c146450 2024-03-25T18:39:15.571 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:15.570+0000 7f97b9810040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:39:15.609 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:15.608+0000 7fe4ca82b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:39:15.637 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.637+0000 7fd611ffb640 1 -- 172.21.15.136:0/2788248085 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "6320cd08-17e0-4bc6-8b8e-9d8341a59cd1", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 3789316321 0 1371665406) 0x7fd614004580 con 0x7fd61c146450 2024-03-25T18:39:15.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.637+0000 7fd611ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "6320cd08-17e0-4bc6-8b8e-9d8341a59cd1", "id": 7}] 2024-03-25T18:39:15.638 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.637+0000 7fd611ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:15.638 INFO:teuthology.orchestra.run.smithi136.stdout:7 2024-03-25T18:39:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 10 librados: watch_flush enter 2024-03-25T18:39:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 10 librados: watch_flush exit 2024-03-25T18:39:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T18:39:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 10 client.4153.objecter successfully canceled tick 2024-03-25T18:39:15.639 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 10 monclient: shutdown 2024-03-25T18:39:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 >> v1:172.21.15.136:6789/0 conn(0x7fd61c146450 legacy=0x7fd61c135380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 shutdown_connections 2024-03-25T18:39:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 >> 172.21.15.136:0/2788248085 conn(0x7fd61c0b7f40 msgr2=0x7fd61c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 shutdown_connections 2024-03-25T18:39:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 1 -- 172.21.15.136:0/2788248085 wait complete. 2024-03-25T18:39:15.640 INFO:teuthology.orchestra.run.smithi136.stderr:2024-03-25T18:39:15.638+0000 7fd6232a5640 1 librados: shutdown 2024-03-25T18:39:15.649 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:39:15.649 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:39:15.650 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:39:15.652 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:39:15.652 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:39:15.652 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:39:15.652 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:39:15.654 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:39:15.654 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:39:15.654 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:39:15.655 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:39:15.657 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:39:15.657 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:39:15.657 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:39:15.657 DEBUG:teuthology.orchestra.run.smithi032:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:39:15.659 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:39:15.659 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:39:15.660 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:39:15.660 DEBUG:teuthology.orchestra.run.smithi136:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:39:15.662 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:39:15.662 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:39:15.662 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:39:15.663 DEBUG:teuthology.orchestra.run.smithi136:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:39:15.665 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:39:15.665 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:39:15.665 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:39:15.665 DEBUG:teuthology.orchestra.run.smithi136:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:39:15.666 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:15.665+0000 7fe4ca82b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:39:15.667 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:39:15.667 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:39:15.668 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:39:15.668 DEBUG:teuthology.orchestra.run.smithi136:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:39:15.670 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:39:15.670 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:15.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.812+0000 7f264bd1c640 10 monclient: get_monmap_and_config 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.812+0000 7f264bd1c640 10 monclient: build_initial_monmap 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.812+0000 7f264bd1c640 10 monclient: monmap: 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:39:15.813135+0000 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:39:15.813135+0000 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:15.813 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.813+0000 7f264bd1c640 1 Processor -- start 2024-03-25T18:39:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.813+0000 7f264bd1c640 1 -- start start 2024-03-25T18:39:15.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.813+0000 7f264bd1c640 10 monclient: init 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.814+0000 7f264bd1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.814+0000 7f264bd1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.814+0000 7f264bd1c640 10 monclient(hunting): picked mon.noname-a con 0x7f26440b4700 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.815+0000 7f264bd1c640 10 monclient(hunting): picked mon.noname-c con 0x7f2644142a20 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.815+0000 7f264bd1c640 10 monclient(hunting): picked mon.noname-b con 0x7f26440a96b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.815+0000 7f264bd1c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26440a9d60 con 0x7f26440b4700 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.815+0000 7f264bd1c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264413ff10 con 0x7f2644142a20 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.815+0000 7f264bd1c640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26440aa1f0 con 0x7f26440a96b0 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.815+0000 7f264bd1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.816+0000 7f264a2b9640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f26440a96b0 0x7f264413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:49842/0 (socket says 172.21.15.32:49842) 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.816+0000 7f264a2b9640 1 -- 172.21.15.32:0/371261291 learned_addr learned my addr 172.21.15.32:0/371261291 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:15.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.816+0000 7f264bd1c640 10 monclient(hunting): authenticate will time out at 6062.004713s 2024-03-25T18:39:15.877 INFO:tasks.ceph.mgr.y.smithi032.stderr:2024-03-25T18:39:15.876+0000 7fe4ca82b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:39:15.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.914+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850842689 0 0) 0x7f26440a9d60 con 0x7f26440b4700 2024-03-25T18:39:15.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.914+0000 7f2648ab6640 10 monclient(hunting): _negotiate 2024-03-25T18:39:15.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.914+0000 7f2648ab6640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:15.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.914+0000 7f2648ab6640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:15.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.914+0000 7f2648ab6640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:39:15.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.914+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2630003520 con 0x7f26440b4700 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1431710745 0 0) 0x7f2630003520 con 0x7f26440b4700 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 >> v1:172.21.15.32:6790/0 conn(0x7f26440a96b0 legacy=0x7f264413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 >> v1:172.21.15.136:6789/0 conn(0x7f2644142a20 legacy=0x7f264413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 10 monclient: _finish_hunting 0 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 1 monclient: found mon.noname-a 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26440b4ae0 con 0x7f26440b4700 2024-03-25T18:39:15.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 10 monclient: _finish_auth 0 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 10 monclient: _check_auth_tickets 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2640001690 con 0x7f26440b4700 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f2648ab6640 10 monclient: dump: 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.921 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.922 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f264bd1c640 5 monclient: authenticate success, global_id 4164 2024-03-25T18:39:15.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.920+0000 7f264bd1c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.923+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f26400017f0 con 0x7f26440b4700 2024-03-25T18:39:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.923+0000 7f2648ab6640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.923+0000 7f2648ab6640 1 -- 172.21.15.32:0/371261291 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2640001c40 con 0x7f26440b4700 2024-03-25T18:39:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.923+0000 7f2648ab6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.923+0000 7f2648ab6640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.923+0000 7f2648ab6640 10 monclient: dump: 2024-03-25T18:39:15.923 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.923+0000 7f264bd1c640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:15.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.923+0000 7f264bd1c640 4 set_mon_vals no callback set 2024-03-25T18:39:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.924+0000 7f264bd1c640 10 monclient: shutdown 2024-03-25T18:39:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.924+0000 7f264bd1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.924+0000 7f264bd1c640 1 -- 172.21.15.32:0/371261291 >> v1:172.21.15.32:6789/0 conn(0x7f26440b4700 legacy=0x7f2644142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.924+0000 7f264bd1c640 1 -- 172.21.15.32:0/371261291 shutdown_connections 2024-03-25T18:39:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.924+0000 7f264bd1c640 1 -- 172.21.15.32:0/371261291 >> 172.21.15.32:0/371261291 conn(0x7f26440668e0 msgr2=0x7f2644066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.924+0000 7f264bd1c640 1 -- 172.21.15.32:0/371261291 shutdown_connections 2024-03-25T18:39:15.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.924+0000 7f264bd1c640 1 -- 172.21.15.32:0/371261291 wait complete. 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.925+0000 7f264bd1c640 10 monclient: build_initial_monmap 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.925+0000 7f264bd1c640 10 monclient: monmap: 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 0 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 0.000000 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:created 0.000000 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 1 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:15.927 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.926+0000 7f264bd1c640 1 librados: starting msgr at 2024-03-25T18:39:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.926+0000 7f264bd1c640 1 librados: starting objecter 2024-03-25T18:39:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.926+0000 7f264bd1c640 1 Processor -- start 2024-03-25T18:39:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.926+0000 7f264bd1c640 1 -- start start 2024-03-25T18:39:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.926+0000 7f264bd1c640 1 librados: setting wanted keys 2024-03-25T18:39:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.926+0000 7f264bd1c640 1 librados: calling monclient init 2024-03-25T18:39:15.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.926+0000 7f264bd1c640 10 monclient: init 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.927+0000 7f264bd1c640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.927+0000 7f264bd1c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.928+0000 7f264bd1c640 10 monclient(hunting): picked mon.noname-a con 0x7f26440a96b0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.928+0000 7f264bd1c640 10 monclient(hunting): picked mon.noname-b con 0x7f26440b4700 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.928+0000 7f264bd1c640 10 monclient(hunting): picked mon.noname-c con 0x7f2644142a20 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.928+0000 7f264bd1c640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26440ad580 con 0x7f26440a96b0 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.928+0000 7f264bd1c640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f264414eca0 con 0x7f26440b4700 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.928+0000 7f264bd1c640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26440ae600 con 0x7f2644142a20 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.928+0000 7f264bd1c640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.928+0000 7f264bd1c640 10 monclient(hunting): authenticate will time out at 6062.117020s 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.929+0000 7f2649ab8640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f26440a96b0 0x7f26440ace70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:40282/0 (socket says 172.21.15.32:40282) 2024-03-25T18:39:15.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.929+0000 7f2649ab8640 1 -- 172.21.15.32:0/1822238161 learned_addr learned my addr 172.21.15.32:0/1822238161 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.930+0000 7f263a7fc640 10 client.?.objecter ms_handle_connect 0x7f26440a96b0 2024-03-25T18:39:15.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.930+0000 7f263a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.933+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 289684758 0 0) 0x7f26440ad580 con 0x7f26440a96b0 2024-03-25T18:39:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.933+0000 7f263a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.933+0000 7f263a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.933+0000 7f263a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.933+0000 7f263a7fc640 10 monclient(hunting): my global_id is 4167 2024-03-25T18:39:15.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.933+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2628004eb0 con 0x7f26440a96b0 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.942+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4181872543 0 0) 0x7f2628004eb0 con 0x7f26440a96b0 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.942+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f26440ad580 con 0x7f26440a96b0 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.942+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2640001690 con 0x7f26440a96b0 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.942+0000 7f263a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.942+0000 7f263a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.942+0000 7f263a7fc640 10 monclient(hunting): dump: 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.942 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.943 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:15.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.942+0000 7f263a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:15.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2899666010 0 0) 0x7f26440ad580 con 0x7f26440a96b0 2024-03-25T18:39:15.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 >> v1:172.21.15.32:6790/0 conn(0x7f2644142a20 legacy=0x7f264413a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 >> v1:172.21.15.136:6789/0 conn(0x7f26440b4700 legacy=0x7f264413a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 1 monclient: found mon.a 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26440ae8e0 con 0x7f26440a96b0 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f263a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f264bd1c640 5 monclient: authenticate success, global_id 4167 2024-03-25T18:39:15.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f264bd1c640 10 monclient: _renew_subs 2024-03-25T18:39:15.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f264bd1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:15.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.947+0000 7f264bd1c640 1 -- 172.21.15.32:0/1822238161 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26440ad8a0 con 0x7f26440a96b0 2024-03-25T18:39:15.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.948+0000 7f264bd1c640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:15.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.948+0000 7f264bd1c640 10 monclient: _renew_subs 2024-03-25T18:39:15.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.948+0000 7f264bd1c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:15.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.948+0000 7f264bd1c640 1 -- 172.21.15.32:0/1822238161 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f26441e3670 con 0x7f26440a96b0 2024-03-25T18:39:15.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.948+0000 7f264bd1c640 1 librados: init done 2024-03-25T18:39:15.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.949+0000 7f262ffff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:15.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.949+0000 7f262ffff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:15.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.949+0000 7f262ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:15.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.949+0000 7f262ffff640 1 -- 172.21.15.32:0/1822238161 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2624005080 con 0x7f26440a96b0 2024-03-25T18:39:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.964+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2640003960 con 0x7f26440a96b0 2024-03-25T18:39:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.964+0000 7f263a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.964+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2640003db0 con 0x7f26440a96b0 2024-03-25T18:39:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.964+0000 7f263a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.965+0000 7f263a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.965+0000 7f263a7fc640 10 monclient: dump: 2024-03-25T18:39:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:epoch 1 2024-03-25T18:39:15.966 INFO:teuthology.orchestra.run.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.965+0000 7f263bfff640 4 set_mon_vals no callback set 2024-03-25T18:39:15.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.965+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 1605001855 0 0) 0x7f2640003fd0 con 0x7f26440a96b0 2024-03-25T18:39:15.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.967+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f26400159b0 con 0x7f26440a96b0 2024-03-25T18:39:15.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.967+0000 7f263a7fc640 10 client.4167.objecter ms_dispatch 0x7f2644000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:39:15.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.967+0000 7f263a7fc640 3 client.4167.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:39:15.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.967+0000 7f263a7fc640 3 client.4167.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:39:15.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:15.967+0000 7f263a7fc640 20 client.4167.objecter dump_active .. 0 homeless 2024-03-25T18:39:16.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.109+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2640014c70 con 0x7f26440a96b0 2024-03-25T18:39:16.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.109+0000 7f263a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:16.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.109+0000 7f263a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.199+0000 7f262ffff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.199+0000 7f262ffff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.199+0000 7f262ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:16.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.199+0000 7f262ffff640 1 -- 172.21.15.32:0/1822238161 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2624005330 con 0x7f26440a96b0 2024-03-25T18:39:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.276+0000 7f263a7fc640 1 -- 172.21.15.32:0/1822238161 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f2640013450 con 0x7f26440a96b0 2024-03-25T18:39:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.276+0000 7f263a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:16.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.276+0000 7f263a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:16.277 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-03-25T18:39:16.277 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:16.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.277+0000 7f262ffff640 10 librados: watch_flush enter 2024-03-25T18:39:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.277+0000 7f262ffff640 10 librados: watch_flush exit 2024-03-25T18:39:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.277+0000 7f262ffff640 20 client.4167.objecter shutdown clearing up homeless session... 2024-03-25T18:39:16.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.277+0000 7f262ffff640 10 client.4167.objecter successfully canceled tick 2024-03-25T18:39:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.277+0000 7f262ffff640 10 monclient: shutdown 2024-03-25T18:39:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.278+0000 7f262ffff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.278+0000 7f262ffff640 1 -- 172.21.15.32:0/1822238161 >> v1:172.21.15.32:6789/0 conn(0x7f26440a96b0 legacy=0x7f26440ace70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.278+0000 7f262ffff640 1 -- 172.21.15.32:0/1822238161 shutdown_connections 2024-03-25T18:39:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.278+0000 7f262ffff640 1 -- 172.21.15.32:0/1822238161 >> 172.21.15.32:0/1822238161 conn(0x7f26440668e0 msgr2=0x7f264413c3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.278+0000 7f262ffff640 1 -- 172.21.15.32:0/1822238161 shutdown_connections 2024-03-25T18:39:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.278+0000 7f262ffff640 1 -- 172.21.15.32:0/1822238161 wait complete. 2024-03-25T18:39:16.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-03-25T18:39:16.278+0000 7f262ffff640 1 librados: shutdown 2024-03-25T18:39:16.294 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:39:16.294 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:39:17.100 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.245+0000 7f9874de6640 10 monclient: get_monmap_and_config 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.245+0000 7f9874de6640 10 monclient: build_initial_monmap 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.245+0000 7f9874de6640 10 monclient: monmap: 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:39:17.246328+0000 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:39:17.246328+0000 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:17.246 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:17.247 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:17.247 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:17.247 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:17.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.246+0000 7f9874de6640 1 Processor -- start 2024-03-25T18:39:17.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.246+0000 7f9874de6640 1 -- start start 2024-03-25T18:39:17.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.246+0000 7f9874de6640 10 monclient: init 2024-03-25T18:39:17.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.248+0000 7f9874de6640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:17.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.248+0000 7f9874de6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:39:17.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.248+0000 7f9874de6640 10 monclient(hunting): picked mon.noname-c con 0x7f98700b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:17.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.248+0000 7f9874de6640 10 monclient(hunting): picked mon.noname-b con 0x7f98700b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:17.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.248+0000 7f9874de6640 10 monclient(hunting): picked mon.noname-a con 0x7f987014be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:17.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.248+0000 7f9874de6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98700af940 con 0x7f987014be90 2024-03-25T18:39:17.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.248+0000 7f9874de6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98700b0a70 con 0x7f98700b4d60 2024-03-25T18:39:17.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.249+0000 7f9874de6640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98700b0d50 con 0x7f98700b1730 2024-03-25T18:39:17.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.249+0000 7f9874de6640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:17.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.249+0000 7f9874de6640 10 monclient(hunting): authenticate will time out at 6063.437728s 2024-03-25T18:39:17.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.249+0000 7f986ed9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f987014be90 0x7f987014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43264/0 (socket says 172.21.15.32:43264) 2024-03-25T18:39:17.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.249+0000 7f986ed9d640 1 -- 172.21.15.32:0/604533971 learned_addr learned my addr 172.21.15.32:0/604533971 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:17.253 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.253+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3632586645 0 0) 0x7f98700af940 con 0x7f987014be90 2024-03-25T18:39:17.253 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.253+0000 7f986d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:39:17.253 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.253+0000 7f986d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:17.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.253+0000 7f986d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:17.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.253+0000 7f986d59a640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:39:17.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.253+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9858003520 con 0x7f987014be90 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2247129304 0 0) 0x7f9858003520 con 0x7f987014be90 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 >> v1:172.21.15.32:6790/0 conn(0x7f98700b1730 legacy=0x7f98700b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 >> v1:172.21.15.136:6789/0 conn(0x7f98700b4d60 legacy=0x7f987013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 1 monclient: found mon.noname-a 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98700b1030 con 0x7f987014be90 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 10 monclient: _finish_auth 0 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f986d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f9874de6640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:39:17.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.258+0000 7f9874de6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.259+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9868001a10 con 0x7f987014be90 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.259+0000 7f986d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.259+0000 7f986d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.259+0000 7f986d59a640 10 monclient: dump: 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:17.261 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:17.262 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.259+0000 7f9874de6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.261+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9868001c70 con 0x7f987014be90 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.261+0000 7f986d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.261+0000 7f986d59a640 1 -- 172.21.15.32:0/604533971 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f98680020c0 con 0x7f987014be90 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.261+0000 7f986d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.261+0000 7f986d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.261+0000 7f986d59a640 10 monclient: dump: 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:17.263 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.261+0000 7f9874de6640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.261+0000 7f9874de6640 4 set_mon_vals no callback set 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.262+0000 7f9874de6640 10 monclient: shutdown 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.262+0000 7f9874de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.262+0000 7f9874de6640 1 -- 172.21.15.32:0/604533971 >> v1:172.21.15.32:6789/0 conn(0x7f987014be90 legacy=0x7f987014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.263+0000 7f9874de6640 1 -- 172.21.15.32:0/604533971 shutdown_connections 2024-03-25T18:39:17.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.263+0000 7f9874de6640 1 -- 172.21.15.32:0/604533971 >> 172.21.15.32:0/604533971 conn(0x7f98700617a0 msgr2=0x7f9870061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:17.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.264+0000 7f9874de6640 1 -- 172.21.15.32:0/604533971 shutdown_connections 2024-03-25T18:39:17.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.264+0000 7f9874de6640 1 -- 172.21.15.32:0/604533971 wait complete. 2024-03-25T18:39:17.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.265+0000 7f9874de6640 10 monclient: build_initial_monmap 2024-03-25T18:39:17.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.265+0000 7f9874de6640 10 monclient: monmap: 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:17.266 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:17.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.266+0000 7f9874de6640 1 librados: starting msgr at 2024-03-25T18:39:17.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.266+0000 7f9874de6640 1 librados: starting objecter 2024-03-25T18:39:17.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.266+0000 7f9874de6640 1 Processor -- start 2024-03-25T18:39:17.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.266+0000 7f9874de6640 1 -- start start 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.266+0000 7f9874de6640 1 librados: setting wanted keys 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.266+0000 7f9874de6640 1 librados: calling monclient init 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.266+0000 7f9874de6640 10 monclient: init 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.267+0000 7f9874de6640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.267+0000 7f9874de6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.267+0000 7f9874de6640 10 monclient(hunting): picked mon.noname-a con 0x7f98700b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.267+0000 7f9874de6640 10 monclient(hunting): picked mon.noname-b con 0x7f98700b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.267+0000 7f9874de6640 10 monclient(hunting): picked mon.noname-c con 0x7f987014be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.268+0000 7f9874de6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987013a2a0 con 0x7f98700b1730 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.268+0000 7f9874de6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9870142fd0 con 0x7f98700b4d60 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.268+0000 7f9874de6640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f987013a4d0 con 0x7f987014be90 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.268+0000 7f9874de6640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:17.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.268+0000 7f9874de6640 10 monclient(hunting): authenticate will time out at 6063.456909s 2024-03-25T18:39:17.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.268+0000 7f986e59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f98700b1730 0x7f98701db230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43280/0 (socket says 172.21.15.32:43280) 2024-03-25T18:39:17.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.268+0000 7f986e59c640 1 -- 172.21.15.32:0/910720398 learned_addr learned my addr 172.21.15.32:0/910720398 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:17.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.269+0000 7f985effd640 10 client.?.objecter ms_handle_connect 0x7f98700b1730 2024-03-25T18:39:17.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.269+0000 7f985effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:17.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.272+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1693600857 0 0) 0x7f987013a2a0 con 0x7f98700b1730 2024-03-25T18:39:17.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.272+0000 7f985effd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:17.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.272+0000 7f985effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:17.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.272+0000 7f985effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:17.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.272+0000 7f985effd640 10 monclient(hunting): my global_id is 4176 2024-03-25T18:39:17.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.272+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9850004eb0 con 0x7f98700b1730 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.280+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1043878082 0 0) 0x7f9850004eb0 con 0x7f98700b1730 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.280+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f987013a2a0 con 0x7f98700b1730 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.280+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9860001960 con 0x7f98700b1730 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.280+0000 7f985effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.280+0000 7f985effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.280+0000 7f985effd640 10 monclient(hunting): dump: 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:17.281 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:17.282 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:17.282 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:17.282 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:17.282 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:17.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.280+0000 7f985effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:17.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3959633217 0 0) 0x7f987013a2a0 con 0x7f98700b1730 2024-03-25T18:39:17.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 >> v1:172.21.15.32:6790/0 conn(0x7f987014be90 legacy=0x7f98701e2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 >> v1:172.21.15.136:6789/0 conn(0x7f98700b4d60 legacy=0x7f9870134db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 10 monclient: _finish_hunting 0 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 1 monclient: found mon.a 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98701db940 con 0x7f98700b1730 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 10 monclient: _finish_auth 0 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 10 monclient: _check_auth_tickets 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.286+0000 7f985effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 5 monclient: authenticate success, global_id 4176 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 10 monclient: _renew_subs 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98701429e0 con 0x7f98700b1730 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 10 client.4176.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:17.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 10 monclient: _renew_subs 2024-03-25T18:39:17.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:17.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9870143400 con 0x7f98700b1730 2024-03-25T18:39:17.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.287+0000 7f9874de6640 1 librados: init done 2024-03-25T18:39:17.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.288+0000 7f9874de6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:17.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.288+0000 7f9874de6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:17.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.288+0000 7f9874de6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:17.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.288+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98700627b0 con 0x7f98700b1730 2024-03-25T18:39:17.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.289+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9860001c80 con 0x7f98700b1730 2024-03-25T18:39:17.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.289+0000 7f985effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:17.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.289+0000 7f986cd99640 4 set_mon_vals no callback set 2024-03-25T18:39:17.318 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.317+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9860003a40 con 0x7f98700b1730 2024-03-25T18:39:17.318 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.317+0000 7f985effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:17.318 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.317+0000 7f985effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:17.318 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.317+0000 7f985effd640 10 monclient: dump: 2024-03-25T18:39:17.318 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:17.319 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.317+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 3) v1 ==== 89499+0+0 (unknown 3666742690 0 0) 0x7f9860003c20 con 0x7f98700b1730 2024-03-25T18:39:17.323 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.323+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f98600547f0 con 0x7f98700b1730 2024-03-25T18:39:17.323 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.323+0000 7f985effd640 10 client.4176.objecter ms_dispatch 0x7f9870000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:39:17.323 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.323+0000 7f985effd640 3 client.4176.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:39:17.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.323+0000 7f985effd640 3 client.4176.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:39:17.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.323+0000 7f985effd640 20 client.4176.objecter dump_active .. 0 homeless 2024-03-25T18:39:17.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.513+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9860050130 con 0x7f98700b1730 2024-03-25T18:39:17.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.513+0000 7f985effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:17.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.513+0000 7f985effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:17.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.641+0000 7f9874de6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:17.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.641+0000 7f9874de6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:17.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.641+0000 7f9874de6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:17.642 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.641+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f987013cc30 con 0x7f98700b1730 2024-03-25T18:39:17.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.666+0000 7f985effd640 1 -- 172.21.15.32:0/910720398 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f9860052230 con 0x7f98700b1730 2024-03-25T18:39:17.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.666+0000 7f985effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:17.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.666+0000 7f985effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:17.668 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:39:17.668 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:17.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 10 librados: watch_flush enter 2024-03-25T18:39:17.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 10 librados: watch_flush exit 2024-03-25T18:39:17.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 20 client.4176.objecter shutdown clearing up homeless session... 2024-03-25T18:39:17.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 10 client.4176.objecter successfully canceled tick 2024-03-25T18:39:17.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 10 monclient: shutdown 2024-03-25T18:39:17.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:17.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 >> v1:172.21.15.32:6789/0 conn(0x7f98700b1730 legacy=0x7f98701db230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:17.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 shutdown_connections 2024-03-25T18:39:17.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 >> 172.21.15.32:0/910720398 conn(0x7f98700617a0 msgr2=0x7f9870061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:17.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 shutdown_connections 2024-03-25T18:39:17.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 1 -- 172.21.15.32:0/910720398 wait complete. 2024-03-25T18:39:17.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:17.668+0000 7f9874de6640 1 librados: shutdown 2024-03-25T18:39:17.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:18.772 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:18.771+0000 7f975a1d5640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:39:18.921 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:39:18.920+0000 7f975a1d5640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:39:22.603 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:06.874 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:22.615 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:06.880 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:22.633 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:06.908 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:22.656 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:06.928 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:22.664 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:06.935 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:22.676 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:06.941 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:22.693 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:06.968 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:22.719 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:06.990 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.217 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-03-25T18:39:23.210+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:39:23.222 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:07.482 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.241 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-03-25T18:39:23.232+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:39:23.243 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-03-25T18:39:23.236+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:39:23.255 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:07.528 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.263 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-03-25T18:39:23.257+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:39:23.276 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:07.522 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.284 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:07.544 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.304 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:07.560 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.317 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:07.590 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.337 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:07.583 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.392 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:07.648 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.703 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:07.968 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.705 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:07.976 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.722 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:07.996 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.759 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:08.030 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.848 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-25T18:39:23.842+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:39:23.859 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:08.134 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:23.873 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-25T18:39:23.867+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:39:23.920 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-25T18:39:23.914+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:39:23.983 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-25T18:39:23.976+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:39:24.079 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:08.343 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.079 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:08.350 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.130 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:08.401 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.220 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:08.495 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.322 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:08.582 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.344 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:08.616 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.393 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:08.640 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.394 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:08.665 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.432 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:08.698 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.454 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:08.710 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.487 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:24.488 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:08.759 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.517 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:08.792 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.576+0000 7f739f7c4640 10 monclient: get_monmap_and_config 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.576+0000 7f739f7c4640 10 monclient: build_initial_monmap 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.576+0000 7f739f7c4640 10 monclient: monmap: 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:39:24.576840+0000 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:39:24.576840+0000 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:24.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.576+0000 7f739f7c4640 1 Processor -- start 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 1 -- start start 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 10 monclient: init 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 10 monclient(hunting): picked mon.noname-a con 0x7f739806ab80 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 10 monclient(hunting): picked mon.noname-b con 0x7f739806af60 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 10 monclient(hunting): picked mon.noname-c con 0x7f73980af580 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980b04d0 con 0x7f739806ab80 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980b1230 con 0x7f73980af580 2024-03-25T18:39:24.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980affb0 con 0x7f739806af60 2024-03-25T18:39:24.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:24.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.577+0000 7f739f7c4640 10 monclient(hunting): authenticate will time out at 6070.766278s 2024-03-25T18:39:24.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.578+0000 7f739cd5f640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f739806af60 0x7f7398142370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:36428/0 (socket says 172.21.15.32:36428) 2024-03-25T18:39:24.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.578+0000 7f739cd5f640 1 -- 172.21.15.32:0/262947531 learned_addr learned my addr 172.21.15.32:0/262947531 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366417831 0 0) 0x7f73980b04d0 con 0x7f739806ab80 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7384003520 con 0x7f739806ab80 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923683799 0 0) 0x7f73980affb0 con 0x7f739806af60 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): my global_id is 4196 2024-03-25T18:39:24.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73980b04d0 con 0x7f739806af60 2024-03-25T18:39:24.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2098929143 0 0) 0x7f73980b1230 con 0x7f73980af580 2024-03-25T18:39:24.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:24.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:24.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:24.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 10 monclient(hunting): my global_id is 4195 2024-03-25T18:39:24.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.580+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73980affb0 con 0x7f73980af580 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3930107650 0 0) 0x7f73980affb0 con 0x7f73980af580 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 >> v1:172.21.15.32:6790/0 conn(0x7f739806af60 legacy=0x7f7398142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 >> v1:172.21.15.32:6789/0 conn(0x7f739806ab80 legacy=0x7f739813f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 1 monclient: found mon.noname-c 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73980b0140 con 0x7f73980af580 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 10 monclient: _finish_auth 0 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:24.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f73900017e0 con 0x7f73980af580 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f738ffff640 10 monclient: dump: 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:24.585 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:24.586 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f739f7c4640 5 monclient: authenticate success, global_id 4195 2024-03-25T18:39:24.586 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.583+0000 7f739f7c4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.585+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7390001940 con 0x7f73980af580 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.585+0000 7f738ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.585+0000 7f738ffff640 1 -- 172.21.15.32:0/262947531 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7390001d10 con 0x7f73980af580 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.585+0000 7f739f7c4640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.585+0000 7f739f7c4640 4 set_mon_vals no callback set 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.585+0000 7f738ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.585+0000 7f738ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.585+0000 7f738ffff640 10 monclient: dump: 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:24.589 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 10 monclient: shutdown 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 1 -- 172.21.15.32:0/262947531 >> v1:172.21.15.136:6789/0 conn(0x7f73980af580 legacy=0x7f739814dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 1 -- 172.21.15.32:0/262947531 shutdown_connections 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 1 -- 172.21.15.32:0/262947531 >> 172.21.15.32:0/262947531 conn(0x7f73980b4b80 msgr2=0x7f73980b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 1 -- 172.21.15.32:0/262947531 shutdown_connections 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 1 -- 172.21.15.32:0/262947531 wait complete. 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 10 monclient: build_initial_monmap 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.586+0000 7f739f7c4640 10 monclient: monmap: 2024-03-25T18:39:24.590 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 librados: starting msgr at 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 librados: starting objecter 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 Processor -- start 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 -- start start 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 librados: setting wanted keys 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 librados: calling monclient init 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 10 monclient: init 2024-03-25T18:39:24.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 10 monclient(hunting): picked mon.noname-b con 0x7f739806ab80 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 10 monclient(hunting): picked mon.noname-c con 0x7f73980af580 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 10 monclient(hunting): picked mon.noname-a con 0x7f73981d7f10 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739811cb30 con 0x7f73981d7f10 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739811f190 con 0x7f739806ab80 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f739811cdd0 con 0x7f73980af580 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.587+0000 7f739f7c4640 10 monclient(hunting): authenticate will time out at 6070.776402s 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f739cd5f640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f73980af580 0x7f739811da00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:36432/0 (socket says 172.21.15.32:36432) 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f739dd61640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f73981d7f10 0x7f73981e1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43324/0 (socket says 172.21.15.32:43324) 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f739cd5f640 1 -- 172.21.15.32:0/1608994912 learned_addr learned my addr 172.21.15.32:0/1608994912 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f739d560640 1 --1- 172.21.15.32:0/1608994912 >> v1:172.21.15.136:6789/0 conn(0x7f739806ab80 0x7f739811d2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:40956/0 (socket says 172.21.15.32:40956) 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f738dffb640 10 client.?.objecter ms_handle_connect 0x7f739806ab80 2024-03-25T18:39:24.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f738dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:24.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f738dffb640 10 client.?.objecter ms_handle_connect 0x7f73981d7f10 2024-03-25T18:39:24.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f738dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:24.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f738dffb640 10 client.?.objecter ms_handle_connect 0x7f73980af580 2024-03-25T18:39:24.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.588+0000 7f738dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:24.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.589+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506450684 0 0) 0x7f739811f190 con 0x7f739806ab80 2024-03-25T18:39:24.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.589+0000 7f738dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:24.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.589+0000 7f738dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:24.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.589+0000 7f738dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:24.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.589+0000 7f738dffb640 10 monclient(hunting): my global_id is 4198 2024-03-25T18:39:24.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.589+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f737c004eb0 con 0x7f739806ab80 2024-03-25T18:39:24.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083340038 0 0) 0x7f739811cb30 con 0x7f73981d7f10 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f739811f190 con 0x7f73981d7f10 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2522786469 0 0) 0x7f739811cdd0 con 0x7f73980af580 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 10 monclient(hunting): my global_id is 4199 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.590+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f739811cb30 con 0x7f73980af580 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.594+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1135904299 0 0) 0x7f737c004eb0 con 0x7f739806ab80 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.594+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f739811cdd0 con 0x7f739806ab80 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.594+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7394001970 con 0x7f739806ab80 2024-03-25T18:39:24.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.594+0000 7f738dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.594+0000 7f738dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.594+0000 7f738dffb640 10 monclient(hunting): dump: 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:24.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.594+0000 7f738dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:24.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.595+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1202239699 0 0) 0x7f739811cb30 con 0x7f73980af580 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.595+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f737c004eb0 con 0x7f73980af580 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.595+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3079927897 0 0) 0x7f739811f190 con 0x7f73981d7f10 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.595+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f739811cb30 con 0x7f73981d7f10 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.595+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7390002070 con 0x7f73981d7f10 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.595+0000 7f738dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.595+0000 7f738dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.595+0000 7f738dffb640 10 monclient(hunting): dump: 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:24.598 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 143738941 0 0) 0x7f739811cdd0 con 0x7f739806ab80 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 >> v1:172.21.15.32:6790/0 conn(0x7f73980af580 legacy=0x7f739811da00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 >> v1:172.21.15.32:6789/0 conn(0x7f73981d7f10 legacy=0x7f73981e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 1 monclient: found mon.b 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f739813c8b0 con 0x7f739806ab80 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 10 monclient: _finish_auth 0 2024-03-25T18:39:24.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f738dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 5 monclient: authenticate success, global_id 4198 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 10 monclient: _renew_subs 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73981d74f0 con 0x7f739806ab80 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 10 client.4198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 10 monclient: _renew_subs 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73981d7890 con 0x7f739806ab80 2024-03-25T18:39:24.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.597+0000 7f739f7c4640 1 librados: init done 2024-03-25T18:39:24.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.599+0000 7f739f7c4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:24.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.599+0000 7f739f7c4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:24.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.599+0000 7f739f7c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:24.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.599+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f739811efb0 con 0x7f739806ab80 2024-03-25T18:39:24.659 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:08.929 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.699 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:08.959 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.720 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:08.986 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.729 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:09.001 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.760 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:09.006 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.762 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:09.034 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.815 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:09.090 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.827 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:09.083 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:24.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.839+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7394003610 con 0x7f739806ab80 2024-03-25T18:39:24.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.839+0000 7f738dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:24.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.839+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7394003a20 con 0x7f739806ab80 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.839+0000 7f738dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.839+0000 7f738dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.839+0000 7f738dffb640 10 monclient: dump: 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:24.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.839+0000 7f738f7fe640 4 set_mon_vals no callback set 2024-03-25T18:39:24.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.840+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f7394003c40 con 0x7f739806ab80 2024-03-25T18:39:24.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.840+0000 7f739cd5f640 10 monclient: get_auth_request con 0x7f737c06e270 auth_method 0 2024-03-25T18:39:24.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.841+0000 7f738dffb640 10 client.4198.objecter ms_handle_connect 0x7f737c06e270 2024-03-25T18:39:24.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.844+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f7394054780 con 0x7f739806ab80 2024-03-25T18:39:24.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.844+0000 7f738dffb640 10 client.4198.objecter ms_dispatch 0x7f7398000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:39:24.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.844+0000 7f738dffb640 3 client.4198.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:39:24.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.844+0000 7f738dffb640 3 client.4198.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:39:24.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.844+0000 7f738dffb640 20 client.4198.objecter dump_active .. 0 homeless 2024-03-25T18:39:24.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.844+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7394050290 con 0x7f739806ab80 2024-03-25T18:39:24.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.844+0000 7f738dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:24.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:24.844+0000 7f738dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:24.923 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:09.193 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.008 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:09.274 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.019 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:09.280 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.024+0000 7f739f7c4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:25.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.024+0000 7f739f7c4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:25.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.024+0000 7f739f7c4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:39:25.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.024+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7398142a00 con 0x7f739806ab80 2024-03-25T18:39:25.065 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:09.337 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.067 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:09.339 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.071+0000 7f738dffb640 1 -- 172.21.15.32:0/1608994912 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f73940521c0 con 0x7f739806ab80 2024-03-25T18:39:25.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.071+0000 7f738dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:25.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.071+0000 7f738dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:25.071 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:39:25.071 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 10 librados: watch_flush enter 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 10 librados: watch_flush exit 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 20 client.4198.objecter shutdown clearing up homeless session... 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 10 client.4198.objecter successfully canceled tick 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 >> v1:172.21.15.136:6800/1141502808 conn(0x7f737c06e270 legacy=0x7f737c070730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 10 monclient: shutdown 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 >> v1:172.21.15.136:6789/0 conn(0x7f739806ab80 legacy=0x7f739811d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 shutdown_connections 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 >> 172.21.15.32:0/1608994912 conn(0x7f73980b4b80 msgr2=0x7f73980b3f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 shutdown_connections 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 1 -- 172.21.15.32:0/1608994912 wait complete. 2024-03-25T18:39:25.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:25.073+0000 7f739f7c4640 1 librados: shutdown 2024-03-25T18:39:25.083 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:25.106 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:09.381 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.116 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:09.388 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.117 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:09.364 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.150 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:09.406 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.188 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:09.459 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.284 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:09.545 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.295 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:09.561 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.354 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:09.626 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.384 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:09.656 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.388 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:09.634 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.395 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:09.670 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.420 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:09.676 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.456 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:09.726 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.549 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:09.810 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.587 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:09.853 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.644 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:09.916 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.683 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:09.929 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.683 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:09.955 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.687 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:09.962 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.688 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:09.944 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.726 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:09.997 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.814 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:10.074 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.875 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:10.141 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.932 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:10.204 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.955 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:10.211 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.977 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:10.223 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.981 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:10.253 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:25.986 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:10.261 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.006 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:10.276 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.078 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:10.339 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.173 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:10.438 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.220 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:10.476 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.222 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:10.494 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.273 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:10.519 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.282 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:10.554 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.346 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:10.607 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.489 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:10.744 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.498 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:10.769 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.526 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:10.801 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.562 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:10.808 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.582 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:10.854 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.587 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:10.858 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.609 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:10.869 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.627 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:10.902 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.721 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:10.986 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.757 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:11.013 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.777 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:11.048 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.819 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:11.085 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.847 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:11.118 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:26.851 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:11.097 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:27.090 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:11.350 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:27.114 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:11.387 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:27.148 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:11.409 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:27.174 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:11.447 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:27.238 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:11.493 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:27.306 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:11.562 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:27.376 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:11.623 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:27.432 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:11.679 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:31.890 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:32.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.023+0000 7f5650227640 10 monclient: get_monmap_and_config 2024-03-25T18:39:32.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.023+0000 7f5650227640 10 monclient: build_initial_monmap 2024-03-25T18:39:32.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.023+0000 7f5650227640 10 monclient: monmap: 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:39:32.024821+0000 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:39:32.024821+0000 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:32.025 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:32.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.024+0000 7f5650227640 1 Processor -- start 2024-03-25T18:39:32.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.024+0000 7f5650227640 1 -- start start 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.024+0000 7f5650227640 10 monclient: init 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 10 monclient(hunting): picked mon.noname-a con 0x7f56480a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 10 monclient(hunting): picked mon.noname-c con 0x7f564813f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 10 monclient(hunting): picked mon.noname-b con 0x7f56481430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5648142520 con 0x7f56480a98f0 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56480b28a0 con 0x7f564813f710 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56480b2a30 con 0x7f56481430c0 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:32.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.025+0000 7f5650227640 10 monclient(hunting): authenticate will time out at 6078.215191s 2024-03-25T18:39:32.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.026+0000 7f564d7c2640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f564813f710 0x7f56481429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43548/0 (socket says 172.21.15.32:43548) 2024-03-25T18:39:32.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.026+0000 7f564d7c2640 1 -- 172.21.15.32:0/325385936 learned_addr learned my addr 172.21.15.32:0/325385936 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.029+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3955838496 0 0) 0x7f56480b2a30 con 0x7f56481430c0 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5634003520 con 0x7f56481430c0 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3229756876 0 0) 0x7f56480b28a0 con 0x7f564813f710 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:39:32.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.030+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56480b2a30 con 0x7f564813f710 2024-03-25T18:39:32.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3934647185 0 0) 0x7f56480b2a30 con 0x7f564813f710 2024-03-25T18:39:32.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 >> v1:172.21.15.32:6790/0 conn(0x7f56481430c0 legacy=0x7f564814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 >> v1:172.21.15.32:6789/0 conn(0x7f56480a98f0 legacy=0x7f56480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:39:32.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 10 monclient: _finish_hunting 0 2024-03-25T18:39:32.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 1 monclient: found mon.noname-c 2024-03-25T18:39:32.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:32.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:39:32.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56480b44c0 con 0x7f564813f710 2024-03-25T18:39:32.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 10 monclient: _finish_auth 0 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 10 monclient: _check_auth_tickets 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5638001670 con 0x7f564813f710 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f564cfc1640 10 monclient: dump: 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:32.038 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f5650227640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:39:32.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.035+0000 7f5650227640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:32.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.037+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56380017d0 con 0x7f564813f710 2024-03-25T18:39:32.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.037+0000 7f564cfc1640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:32.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.037+0000 7f564cfc1640 1 -- 172.21.15.32:0/325385936 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5638001c20 con 0x7f564813f710 2024-03-25T18:39:32.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.037+0000 7f5650227640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.037+0000 7f5650227640 4 set_mon_vals no callback set 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.037+0000 7f564cfc1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.037+0000 7f564cfc1640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.037+0000 7f564cfc1640 10 monclient: dump: 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:32.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.038+0000 7f5650227640 10 monclient: shutdown 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.038+0000 7f5650227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.038+0000 7f5650227640 1 -- 172.21.15.32:0/325385936 >> v1:172.21.15.136:6789/0 conn(0x7f564813f710 legacy=0x7f56481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.038+0000 7f5650227640 1 -- 172.21.15.32:0/325385936 shutdown_connections 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.038+0000 7f5650227640 1 -- 172.21.15.32:0/325385936 >> 172.21.15.32:0/325385936 conn(0x7f56480b7ea0 msgr2=0x7f56480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.038+0000 7f5650227640 1 -- 172.21.15.32:0/325385936 shutdown_connections 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.039+0000 7f5650227640 1 -- 172.21.15.32:0/325385936 wait complete. 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.039+0000 7f5650227640 10 monclient: build_initial_monmap 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.039+0000 7f5650227640 10 monclient: monmap: 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:39:32.042 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:32.043 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:39:32.043 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:32.043 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:32.043 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:32.043 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.040+0000 7f5650227640 1 librados: starting msgr at 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.040+0000 7f5650227640 1 librados: starting objecter 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.040+0000 7f5650227640 1 Processor -- start 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.040+0000 7f5650227640 1 -- start start 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.040+0000 7f5650227640 1 librados: setting wanted keys 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.040+0000 7f5650227640 1 librados: calling monclient init 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.040+0000 7f5650227640 10 monclient: init 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 10 monclient(hunting): picked mon.noname-c con 0x7f56480a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:32.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 10 monclient(hunting): picked mon.noname-a con 0x7f564813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 10 monclient(hunting): picked mon.noname-b con 0x7f56481430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56480abf80 con 0x7f564813f710 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56481d51f0 con 0x7f56481430c0 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56481d7730 con 0x7f56480a98f0 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.041+0000 7f5650227640 10 monclient(hunting): authenticate will time out at 6078.231293s 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.042+0000 7f564dfc3640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f56480a98f0 0x7f5648137e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:56170/0 (socket says 172.21.15.32:56170) 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.042+0000 7f564dfc3640 1 -- 172.21.15.32:0/3332157424 learned_addr learned my addr 172.21.15.32:0/3332157424 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.042+0000 7f563e7fc640 10 client.?.objecter ms_handle_connect 0x7f56480a98f0 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.042+0000 7f563e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.042+0000 7f563e7fc640 10 client.?.objecter ms_handle_connect 0x7f564813f710 2024-03-25T18:39:32.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.042+0000 7f563e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.044+0000 7f563e7fc640 10 client.?.objecter ms_handle_connect 0x7f56481430c0 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.044+0000 7f563e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.045+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865444278 0 0) 0x7f56481d7730 con 0x7f56480a98f0 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.045+0000 7f563e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.045+0000 7f563e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.045+0000 7f563e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.045+0000 7f563e7fc640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.045+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f562c004eb0 con 0x7f56480a98f0 2024-03-25T18:39:32.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.047+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2935018687 0 0) 0x7f56480abf80 con 0x7f564813f710 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.047+0000 7f563e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.047+0000 7f563e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.047+0000 7f563e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.047+0000 7f563e7fc640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.047+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56481d7730 con 0x7f564813f710 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.048+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327406011 0 0) 0x7f56481d51f0 con 0x7f56481430c0 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.048+0000 7f563e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.048+0000 7f563e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:32.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.048+0000 7f563e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:32.050 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.048+0000 7f563e7fc640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:39:32.050 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.048+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f56480abf80 con 0x7f56481430c0 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.053+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1592028651 0 0) 0x7f562c004eb0 con 0x7f56480a98f0 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.053+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56481d51f0 con 0x7f56480a98f0 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.053+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5644001960 con 0x7f56480a98f0 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.053+0000 7f563e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.053+0000 7f563e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.053+0000 7f563e7fc640 10 monclient(hunting): dump: 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.055 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.056 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.056 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:32.056 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:32.056 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:32.056 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:32.056 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:32.056 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.053+0000 7f563e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:32.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.054+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3423907236 0 0) 0x7f56481d7730 con 0x7f564813f710 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.054+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f562c004eb0 con 0x7f564813f710 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1436524987 0 0) 0x7f56480abf80 con 0x7f56481430c0 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f56481d7730 con 0x7f56481430c0 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5638001b80 con 0x7f564813f710 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 10 monclient(hunting): dump: 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.058 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5640001bd0 con 0x7f56481430c0 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.055+0000 7f563e7fc640 10 monclient(hunting): dump: 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.059 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.060 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:32.060 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:32.060 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:32.060 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:32.060 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:32.061 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3385556415 0 0) 0x7f56481d51f0 con 0x7f56480a98f0 2024-03-25T18:39:32.061 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 >> v1:172.21.15.136:6789/0 conn(0x7f56481430c0 legacy=0x7f56481e1dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.061 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 >> v1:172.21.15.32:6789/0 conn(0x7f564813f710 legacy=0x7f56481d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 1 monclient: found mon.c 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56481d8360 con 0x7f56480a98f0 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f563e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 5 monclient: authenticate success, global_id 4205 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 10 monclient: _renew_subs 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5648138bd0 con 0x7f56480a98f0 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 10 client.4205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:32.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 10 monclient: _renew_subs 2024-03-25T18:39:32.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:32.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f56481e34e0 con 0x7f56480a98f0 2024-03-25T18:39:32.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.059+0000 7f5650227640 1 librados: init done 2024-03-25T18:39:32.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.060+0000 7f5650227640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:32.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.060+0000 7f5650227640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:32.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.060+0000 7f5650227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:32.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.060+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56481d7920 con 0x7f56480a98f0 2024-03-25T18:39:32.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.062+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f56440035f0 con 0x7f56480a98f0 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.062+0000 7f563e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.062+0000 7f563ffff640 4 set_mon_vals no callback set 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.063+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5644003940 con 0x7f56480a98f0 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.063+0000 7f563e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.063+0000 7f563e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.063+0000 7f563e7fc640 10 monclient: dump: 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:32.064 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.065 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:32.065 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:32.065 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:32.065 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:32.065 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:32.065 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:32.065 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:32.091 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.090+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5644003b20 con 0x7f56480a98f0 2024-03-25T18:39:32.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.091+0000 7f564d7c2640 10 monclient: get_auth_request con 0x7f562c06e0e0 auth_method 0 2024-03-25T18:39:32.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.091+0000 7f563e7fc640 10 client.4205.objecter ms_handle_connect 0x7f562c06e0e0 2024-03-25T18:39:32.137 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:16.408 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:32.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.453+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5644054800 con 0x7f56480a98f0 2024-03-25T18:39:32.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.453+0000 7f563e7fc640 10 client.4205.objecter ms_dispatch 0x7f5648000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:39:32.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.453+0000 7f563e7fc640 3 client.4205.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:39:32.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.453+0000 7f563e7fc640 3 client.4205.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:39:32.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.453+0000 7f563e7fc640 20 client.4205.objecter dump_active .. 0 homeless 2024-03-25T18:39:32.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.453+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5644050310 con 0x7f56480a98f0 2024-03-25T18:39:32.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.453+0000 7f563e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:32.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.453+0000 7f563e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:32.476 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:16.751 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:32.479 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:16.750 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:32.544 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:16.809 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:32.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.635+0000 7f5650227640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:32.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.635+0000 7f5650227640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:32.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.635+0000 7f5650227640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:32.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.635+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f56480627b0 con 0x7f56480a98f0 2024-03-25T18:39:32.659 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:16.932 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:32.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.705+0000 7f563e7fc640 1 -- 172.21.15.32:0/3332157424 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5644052240 con 0x7f56480a98f0 2024-03-25T18:39:32.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.705+0000 7f563e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:32.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.705+0000 7f563e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:32.706 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:39:32.706 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:32.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 10 librados: watch_flush enter 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 10 librados: watch_flush exit 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 20 client.4205.objecter shutdown clearing up homeless session... 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 10 client.4205.objecter successfully canceled tick 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 >> v1:172.21.15.136:6800/1141502808 conn(0x7f562c06e0e0 legacy=0x7f562c0705a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 10 monclient: shutdown 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 >> v1:172.21.15.32:6790/0 conn(0x7f56480a98f0 legacy=0x7f5648137e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 shutdown_connections 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 >> 172.21.15.32:0/3332157424 conn(0x7f56480b7ea0 msgr2=0x7f5648074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 shutdown_connections 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 1 -- 172.21.15.32:0/3332157424 wait complete. 2024-03-25T18:39:32.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:32.707+0000 7f5650227640 1 librados: shutdown 2024-03-25T18:39:32.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:32.934 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:17.181 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:33.075 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:17.335 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:33.325 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:17.581 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:37.973 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:22.246 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:37.977 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:22.250 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.033 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:22.298 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.037 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:22.302 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.087 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:22.361 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.091 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:22.365 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.172 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:22.443 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.175 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:22.447 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.378 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:22.648 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.381 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:22.652 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.586 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:22.833 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.590 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:22.837 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.822 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:23.082 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:38.826 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:23.086 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:39.044 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:23.300 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:39.048 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:23.304 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:39.526 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:39.618 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.614+0000 7f4d8f8cf640 10 monclient: get_monmap_and_config 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.614+0000 7f4d8f8cf640 10 monclient: build_initial_monmap 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.614+0000 7f4d8f8cf640 10 monclient: monmap: 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:39:39.616029+0000 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:39:39.616029+0000 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:39.619 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:39.620 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:39.620 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:39.620 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:39.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.615+0000 7f4d8f8cf640 1 Processor -- start 2024-03-25T18:39:39.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.615+0000 7f4d8f8cf640 1 -- start start 2024-03-25T18:39:39.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.615+0000 7f4d8f8cf640 10 monclient: init 2024-03-25T18:39:39.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:39.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:39:39.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 10 monclient(hunting): picked mon.noname-a con 0x7f4d880b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:39.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 10 monclient(hunting): picked mon.noname-b con 0x7f4d880b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 10 monclient(hunting): picked mon.noname-c con 0x7f4d8814be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d880af940 con 0x7f4d880b4d60 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d880b0a70 con 0x7f4d8814be90 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d880b0d50 con 0x7f4d880b1730 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.616+0000 7f4d8f8cf640 10 monclient(hunting): authenticate will time out at 6085.806378s 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.617+0000 7f4d8ce6a640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f4d880b1730 0x7f4d880b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:60292/0 (socket says 172.21.15.32:60292) 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.617+0000 7f4d8ce6a640 1 -- 172.21.15.32:0/3287956952 learned_addr learned my addr 172.21.15.32:0/3287956952 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912781293 0 0) 0x7f4d880b0a70 con 0x7f4d8814be90 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:39.623 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d74003520 con 0x7f4d8814be90 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297368237 0 0) 0x7f4d880b0d50 con 0x7f4d880b1730 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d880b0a70 con 0x7f4d880b1730 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488744398 0 0) 0x7f4d880af940 con 0x7f4d880b4d60 2024-03-25T18:39:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:39.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:39.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:39.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:39:39.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.620+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d880b0d50 con 0x7f4d880b4d60 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 905609402 0 0) 0x7f4d74003520 con 0x7f4d8814be90 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 >> v1:172.21.15.32:6790/0 conn(0x7f4d880b1730 legacy=0x7f4d880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 >> v1:172.21.15.32:6789/0 conn(0x7f4d880b4d60 legacy=0x7f4d8813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 1 monclient: found mon.noname-c 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d880b1030 con 0x7f4d8814be90 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 10 monclient: _finish_auth 0 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:39.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4d80001a10 con 0x7f4d8814be90 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d8f8cf640 5 monclient: authenticate success, global_id 4207 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d7ffff640 10 monclient: dump: 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:39.629 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:39.630 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:39.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.623+0000 7f4d8f8cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:39.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.625+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d80001b70 con 0x7f4d8814be90 2024-03-25T18:39:39.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.625+0000 7f4d7ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:39.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.625+0000 7f4d7ffff640 1 -- 172.21.15.32:0/3287956952 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4d80001f80 con 0x7f4d8814be90 2024-03-25T18:39:39.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.625+0000 7f4d7ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:39.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.625+0000 7f4d7ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.625+0000 7f4d7ffff640 10 monclient: dump: 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:39.634 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.625+0000 7f4d8f8cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.625+0000 7f4d8f8cf640 4 set_mon_vals no callback set 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.626+0000 7f4d8f8cf640 10 monclient: shutdown 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.626+0000 7f4d8f8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.626+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/3287956952 >> v1:172.21.15.136:6789/0 conn(0x7f4d8814be90 legacy=0x7f4d8814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.627+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/3287956952 shutdown_connections 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.627+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/3287956952 >> 172.21.15.32:0/3287956952 conn(0x7f4d880617a0 msgr2=0x7f4d88061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.628+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/3287956952 shutdown_connections 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.628+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/3287956952 wait complete. 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.629+0000 7f4d8f8cf640 10 monclient: build_initial_monmap 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.629+0000 7f4d8f8cf640 10 monclient: monmap: 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:39.635 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.629+0000 7f4d8f8cf640 1 librados: starting msgr at 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.629+0000 7f4d8f8cf640 1 librados: starting objecter 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.630+0000 7f4d8f8cf640 1 Processor -- start 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.630+0000 7f4d8f8cf640 1 -- start start 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.630+0000 7f4d8f8cf640 1 librados: setting wanted keys 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.630+0000 7f4d8f8cf640 1 librados: calling monclient init 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.630+0000 7f4d8f8cf640 10 monclient: init 2024-03-25T18:39:39.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.631+0000 7f4d8f8cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.631+0000 7f4d8f8cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.631+0000 7f4d8f8cf640 10 monclient(hunting): picked mon.noname-a con 0x7f4d880b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.631+0000 7f4d8f8cf640 10 monclient(hunting): picked mon.noname-b con 0x7f4d880b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.631+0000 7f4d8f8cf640 10 monclient(hunting): picked mon.noname-c con 0x7f4d8814be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.631+0000 7f4d8f8cf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8806ee70 con 0x7f4d880b1730 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.631+0000 7f4d8f8cf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d8806f150 con 0x7f4d880b4d60 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.631+0000 7f4d8f8cf640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d881407a0 con 0x7f4d8814be90 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d8f8cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d8f8cf640 10 monclient(hunting): authenticate will time out at 6085.821590s 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d8ce6a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4d880b4d60 0x7f4d88140cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:37802/0 (socket says 172.21.15.32:37802) 2024-03-25T18:39:39.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d8ce6a640 1 -- 172.21.15.32:0/25974973 learned_addr learned my addr 172.21.15.32:0/25974973 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:39.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d7dffb640 10 client.?.objecter ms_handle_connect 0x7f4d8814be90 2024-03-25T18:39:39.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:39.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d7dffb640 10 client.?.objecter ms_handle_connect 0x7f4d880b4d60 2024-03-25T18:39:39.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:39.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d7dffb640 10 client.?.objecter ms_handle_connect 0x7f4d880b1730 2024-03-25T18:39:39.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.632+0000 7f4d7dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.634+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1638549470 0 0) 0x7f4d881407a0 con 0x7f4d8814be90 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d6c004eb0 con 0x7f4d8814be90 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263083795 0 0) 0x7f4d8806f150 con 0x7f4d880b4d60 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:39.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:39.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:39:39.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d881407a0 con 0x7f4d880b4d60 2024-03-25T18:39:39.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000848974 0 0) 0x7f4d8806ee70 con 0x7f4d880b1730 2024-03-25T18:39:39.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:39.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:39.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:39.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:39:39.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.635+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d8806f150 con 0x7f4d880b1730 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.641+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549123690 0 0) 0x7f4d6c004eb0 con 0x7f4d8814be90 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.641+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d8806ee70 con 0x7f4d8814be90 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.641+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3258814864 0 0) 0x7f4d881407a0 con 0x7f4d880b4d60 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.641+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d6c004eb0 con 0x7f4d880b4d60 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.642+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4d80001e80 con 0x7f4d8814be90 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.642+0000 7f4d7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.642+0000 7f4d7dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.642+0000 7f4d7dffb640 10 monclient(hunting): dump: 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:39.643 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:39.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.642+0000 7f4d7dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4d78001910 con 0x7f4d880b4d60 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 10 monclient(hunting): dump: 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:39.645 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 713660145 0 0) 0x7f4d8806f150 con 0x7f4d880b1730 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4d881407a0 con 0x7f4d880b1730 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4d84001870 con 0x7f4d880b1730 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.643+0000 7f4d7dffb640 10 monclient(hunting): dump: 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:39.646 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:39.647 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.647 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.647 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:39.647 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:39.647 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:39.647 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:39.647 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:39.647 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1865375819 0 0) 0x7f4d8806ee70 con 0x7f4d8814be90 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 >> v1:172.21.15.136:6789/0 conn(0x7f4d880b4d60 legacy=0x7f4d88140cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 >> v1:172.21.15.32:6789/0 conn(0x7f4d880b1730 legacy=0x7f4d88142510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 1 monclient: found mon.c 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d88140380 con 0x7f4d8814be90 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 10 monclient: _finish_auth 0 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d7dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:39.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 5 monclient: authenticate success, global_id 4211 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 10 monclient: _renew_subs 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d8806de80 con 0x7f4d8814be90 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 10 client.4211.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 10 monclient: _renew_subs 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d8814f0d0 con 0x7f4d8814be90 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.646+0000 7f4d8f8cf640 1 librados: init done 2024-03-25T18:39:39.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.647+0000 7f4d8f8cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:39.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.647+0000 7f4d8f8cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:39.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.647+0000 7f4d8f8cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:39.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.647+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d881405f0 con 0x7f4d8814be90 2024-03-25T18:39:39.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.649+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4d80002020 con 0x7f4d8814be90 2024-03-25T18:39:39.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.649+0000 7f4d7dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:39.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.649+0000 7f4d7f7fe640 4 set_mon_vals no callback set 2024-03-25T18:39:39.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.675+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4d80003dc0 con 0x7f4d8814be90 2024-03-25T18:39:39.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.675+0000 7f4d7dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:39.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.675+0000 7f4d7dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:39.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.675+0000 7f4d7dffb640 10 monclient: dump: 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:39.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.676+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f4d80019d90 con 0x7f4d8814be90 2024-03-25T18:39:39.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.677+0000 7f4d8d66b640 10 monclient: get_auth_request con 0x7f4d6c06e090 auth_method 0 2024-03-25T18:39:39.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.678+0000 7f4d7dffb640 10 client.4211.objecter ms_handle_connect 0x7f4d6c06e090 2024-03-25T18:39:39.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.680+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f4d80053850 con 0x7f4d8814be90 2024-03-25T18:39:39.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.680+0000 7f4d7dffb640 10 client.4211.objecter ms_dispatch 0x7f4d88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:39:39.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.680+0000 7f4d7dffb640 3 client.4211.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:39:39.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.680+0000 7f4d7dffb640 3 client.4211.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:39:39.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:39.680+0000 7f4d7dffb640 20 client.4211.objecter dump_active .. 0 homeless 2024-03-25T18:39:40.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.116+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4d8004f360 con 0x7f4d8814be90 2024-03-25T18:39:40.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.116+0000 7f4d7dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:40.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.116+0000 7f4d7dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:40.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.425+0000 7f4d8f8cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:40.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.425+0000 7f4d8f8cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:40.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.425+0000 7f4d8f8cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:40.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.425+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4d880627b0 con 0x7f4d8814be90 2024-03-25T18:39:40.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.450+0000 7f4d7dffb640 1 -- 172.21.15.32:0/25974973 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f4d80051290 con 0x7f4d8814be90 2024-03-25T18:39:40.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.450+0000 7f4d7dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:40.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.450+0000 7f4d7dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:40.452 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:39:40.452 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:40.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.453+0000 7f4d8f8cf640 10 librados: watch_flush enter 2024-03-25T18:39:40.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.453+0000 7f4d8f8cf640 10 librados: watch_flush exit 2024-03-25T18:39:40.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.453+0000 7f4d8f8cf640 20 client.4211.objecter shutdown clearing up homeless session... 2024-03-25T18:39:40.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.453+0000 7f4d8f8cf640 10 client.4211.objecter successfully canceled tick 2024-03-25T18:39:40.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 >> v1:172.21.15.136:6800/1141502808 conn(0x7f4d6c06e090 legacy=0x7f4d6c070550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:40.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 10 monclient: shutdown 2024-03-25T18:39:40.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:40.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 >> v1:172.21.15.32:6790/0 conn(0x7f4d8814be90 legacy=0x7f4d881e68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:40.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 shutdown_connections 2024-03-25T18:39:40.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 >> 172.21.15.32:0/25974973 conn(0x7f4d880617a0 msgr2=0x7f4d880b6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:40.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 shutdown_connections 2024-03-25T18:39:40.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 1 -- 172.21.15.32:0/25974973 wait complete. 2024-03-25T18:39:40.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:40.457+0000 7f4d8f8cf640 1 librados: shutdown 2024-03-25T18:39:40.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:42.228 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:00:26.498 35407== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:42.251 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:00:26.524 35183== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:42.381 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:00:26.646 35408== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:42.500 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:00:26.775 35405== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:42.603 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:00:26.874 35406== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:43.105 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:00:27.352 35186== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:43.423 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:00:27.679 35185== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:43.430 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:00:27.691 35184== Warning: unimplemented fcntl command: 1038 2024-03-25T18:39:47.278 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:47.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.398+0000 7f359b8c8640 10 monclient: get_monmap_and_config 2024-03-25T18:39:47.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.398+0000 7f359b8c8640 10 monclient: build_initial_monmap 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.398+0000 7f359b8c8640 10 monclient: monmap: 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:39:47.400089+0000 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:39:47.400089+0000 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:47.400 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.399+0000 7f359b8c8640 1 Processor -- start 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.399+0000 7f359b8c8640 1 -- start start 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.399+0000 7f359b8c8640 10 monclient: init 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.399+0000 7f359b8c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.399+0000 7f359b8c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f359b8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f35940b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f359b8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f35940b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f359b8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f359414be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f359b8c8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940af940 con 0x7f35940b1730 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f359b8c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940b0a70 con 0x7f359414be90 2024-03-25T18:39:47.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f359b8c8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35940b0d50 con 0x7f35940b4d60 2024-03-25T18:39:47.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f359b8c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:47.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f359b8c8640 10 monclient(hunting): authenticate will time out at 6093.589981s 2024-03-25T18:39:47.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f3599664640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f35940b4d60 0x7f359413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:44664/0 (socket says 172.21.15.32:44664) 2024-03-25T18:39:47.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f3599664640 1 -- 172.21.15.32:0/1495724940 learned_addr learned my addr 172.21.15.32:0/1495724940 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:47.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.400+0000 7f3599e65640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f359414be90 0x7f359414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:53312/0 (socket says 172.21.15.32:53312) 2024-03-25T18:39:47.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.403+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1856374982 0 0) 0x7f35940b0d50 con 0x7f35940b4d60 2024-03-25T18:39:47.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.403+0000 7f358bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:47.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.403+0000 7f358bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:47.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.403+0000 7f358bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:47.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.403+0000 7f358bfff640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:39:47.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.403+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3580003520 con 0x7f35940b4d60 2024-03-25T18:39:47.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2978325050 0 0) 0x7f35940b0a70 con 0x7f359414be90 2024-03-25T18:39:47.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:47.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:47.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:47.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:39:47.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35940b0d50 con 0x7f359414be90 2024-03-25T18:39:47.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223208778 0 0) 0x7f35940af940 con 0x7f35940b1730 2024-03-25T18:39:47.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:39:47.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:47.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:47.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:39:47.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.404+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35940b0a70 con 0x7f35940b1730 2024-03-25T18:39:47.408 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.406+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1669540008 0 0) 0x7f3580003520 con 0x7f35940b4d60 2024-03-25T18:39:47.408 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 >> v1:172.21.15.136:6789/0 conn(0x7f359414be90 legacy=0x7f359414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:47.408 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 >> v1:172.21.15.32:6789/0 conn(0x7f35940b1730 legacy=0x7f35940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:47.408 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:39:47.408 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 1 monclient: found mon.noname-b 2024-03-25T18:39:47.408 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35940b1030 con 0x7f35940b4d60 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 10 monclient: _finish_auth 0 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f359b8c8640 5 monclient: authenticate success, global_id 4214 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f359b8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f3590001690 con 0x7f35940b4d60 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f358bfff640 10 monclient: dump: 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:47.409 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.410 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.410 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:47.410 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:47.410 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:47.410 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:47.410 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:47.410 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:47.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.407+0000 7f359b8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.408+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3590001870 con 0x7f35940b4d60 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.408+0000 7f358bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.408+0000 7f358bfff640 1 -- 172.21.15.32:0/1495724940 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f3590001690 con 0x7f35940b4d60 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.408+0000 7f358bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.408+0000 7f358bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.408+0000 7f358bfff640 10 monclient: dump: 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.411 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.408+0000 7f359b8c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.408+0000 7f359b8c8640 4 set_mon_vals no callback set 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.409+0000 7f359b8c8640 10 monclient: shutdown 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.409+0000 7f359b8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.409+0000 7f359b8c8640 1 -- 172.21.15.32:0/1495724940 >> v1:172.21.15.32:6790/0 conn(0x7f35940b4d60 legacy=0x7f359413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.409+0000 7f359b8c8640 1 -- 172.21.15.32:0/1495724940 shutdown_connections 2024-03-25T18:39:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.409+0000 7f359b8c8640 1 -- 172.21.15.32:0/1495724940 >> 172.21.15.32:0/1495724940 conn(0x7f35940617a0 msgr2=0x7f3594061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:47.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.410+0000 7f359b8c8640 1 -- 172.21.15.32:0/1495724940 shutdown_connections 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.410+0000 7f359b8c8640 1 -- 172.21.15.32:0/1495724940 wait complete. 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.411+0000 7f359b8c8640 10 monclient: build_initial_monmap 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.411+0000 7f359b8c8640 10 monclient: monmap: 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.412+0000 7f359b8c8640 1 librados: starting msgr at 2024-03-25T18:39:47.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.412+0000 7f359b8c8640 1 librados: starting objecter 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.412+0000 7f359b8c8640 1 Processor -- start 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.412+0000 7f359b8c8640 1 -- start start 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.412+0000 7f359b8c8640 1 librados: setting wanted keys 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.412+0000 7f359b8c8640 1 librados: calling monclient init 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.412+0000 7f359b8c8640 10 monclient: init 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.413+0000 7f359b8c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.413+0000 7f359b8c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.413+0000 7f359b8c8640 10 monclient(hunting): picked mon.noname-a con 0x7f35940b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.413+0000 7f359b8c8640 10 monclient(hunting): picked mon.noname-c con 0x7f35940b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.413+0000 7f359b8c8640 10 monclient(hunting): picked mon.noname-b con 0x7f359414be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.413+0000 7f359b8c8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35941339d0 con 0x7f35940b1730 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.413+0000 7f359b8c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35941daf10 con 0x7f359414be90 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f359b8c8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35941dbb00 con 0x7f35940b4d60 2024-03-25T18:39:47.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f359b8c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f359b8c8640 10 monclient(hunting): authenticate will time out at 6093.603704s 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f3599664640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f35940b1730 0x7f35941db3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51198/0 (socket says 172.21.15.32:51198) 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f3599664640 1 -- 172.21.15.32:0/2012258281 learned_addr learned my addr 172.21.15.32:0/2012258281 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f3589ffb640 10 client.?.objecter ms_handle_connect 0x7f35940b1730 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f3589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f3589ffb640 10 client.?.objecter ms_handle_connect 0x7f359414be90 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.414+0000 7f3589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.415+0000 7f3589ffb640 10 client.?.objecter ms_handle_connect 0x7f35940b4d60 2024-03-25T18:39:47.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.415+0000 7f3589ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:47.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.417+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3430444103 0 0) 0x7f35941339d0 con 0x7f35940b1730 2024-03-25T18:39:47.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.417+0000 7f3589ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:47.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.417+0000 7f3589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:47.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.417+0000 7f3589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:47.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.417+0000 7f3589ffb640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:39:47.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.417+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3578004eb0 con 0x7f35940b1730 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2259856321 0 0) 0x7f35941daf10 con 0x7f359414be90 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35941339d0 con 0x7f359414be90 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339987289 0 0) 0x7f35941dbb00 con 0x7f35940b4d60 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:39:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.418+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35941daf10 con 0x7f35940b4d60 2024-03-25T18:39:47.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.423+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2639543588 0 0) 0x7f3578004eb0 con 0x7f35940b1730 2024-03-25T18:39:47.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.423+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35941dbb00 con 0x7f35940b1730 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.423+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f35900047e0 con 0x7f35940b1730 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.423+0000 7f3589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.423+0000 7f3589ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.423+0000 7f3589ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:47.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.423+0000 7f3589ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:47.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.425+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 576986133 0 0) 0x7f35941339d0 con 0x7f359414be90 2024-03-25T18:39:47.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.425+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3578004eb0 con 0x7f359414be90 2024-03-25T18:39:47.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.425+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f358c001a90 con 0x7f359414be90 2024-03-25T18:39:47.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.425+0000 7f3589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:47.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.425+0000 7f3589ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:47.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.425+0000 7f3589ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:47.427 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.426+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2305560501 0 0) 0x7f35941daf10 con 0x7f35940b4d60 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.426+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35941339d0 con 0x7f35940b4d60 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.426+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f3584001830 con 0x7f35940b4d60 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.426+0000 7f3589ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:47.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.426+0000 7f3589ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.426+0000 7f3589ffb640 10 monclient(hunting): dump: 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:47.429 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1317931594 0 0) 0x7f35941dbb00 con 0x7f35940b1730 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 >> v1:172.21.15.32:6790/0 conn(0x7f35940b4d60 legacy=0x7f359413b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 >> v1:172.21.15.136:6789/0 conn(0x7f359414be90 legacy=0x7f35941d7aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 1 monclient: found mon.a 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35941dbc90 con 0x7f35940b1730 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 10 monclient: _finish_auth 0 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f3589ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 10 monclient: _renew_subs 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3594136530 con 0x7f35940b1730 2024-03-25T18:39:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 10 monclient: _renew_subs 2024-03-25T18:39:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f359413bf80 con 0x7f35940b1730 2024-03-25T18:39:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.428+0000 7f359b8c8640 1 librados: init done 2024-03-25T18:39:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.429+0000 7f359b8c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.429+0000 7f359b8c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.429+0000 7f359b8c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.430+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35940627b0 con 0x7f35940b1730 2024-03-25T18:39:47.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.767+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3590004bf0 con 0x7f35940b1730 2024-03-25T18:39:47.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.767+0000 7f3589ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.767+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f3590004f00 con 0x7f35940b1730 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.767+0000 7f3589ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.767+0000 7f3589ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.767+0000 7f3589ffb640 10 monclient: dump: 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:47.769 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:47.770 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:47.770 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:47.770 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.767+0000 7f358b7fe640 4 set_mon_vals no callback set 2024-03-25T18:39:47.771 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.768+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f359001aed0 con 0x7f35940b1730 2024-03-25T18:39:47.771 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.769+0000 7f3598e63640 10 monclient: get_auth_request con 0x7f357806e0b0 auth_method 0 2024-03-25T18:39:47.771 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.769+0000 7f3589ffb640 10 client.4221.objecter ms_handle_connect 0x7f357806e0b0 2024-03-25T18:39:47.777 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.776+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f35900549f0 con 0x7f35940b1730 2024-03-25T18:39:47.777 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.776+0000 7f3589ffb640 10 client.4221.objecter ms_dispatch 0x7f3594000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:39:47.777 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.776+0000 7f3589ffb640 3 client.4221.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:39:47.777 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.776+0000 7f3589ffb640 3 client.4221.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:39:47.777 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.776+0000 7f3589ffb640 20 client.4221.objecter dump_active .. 0 homeless 2024-03-25T18:39:47.777 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.776+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3590050500 con 0x7f35940b1730 2024-03-25T18:39:47.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.776+0000 7f3589ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:47.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.776+0000 7f3589ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:47.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.963+0000 7f359b8c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:47.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.963+0000 7f359b8c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:47.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.963+0000 7f359b8c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:39:47.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.963+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3594069f40 con 0x7f35940b1730 2024-03-25T18:39:47.979 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.978+0000 7f3589ffb640 1 -- 172.21.15.32:0/2012258281 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f3590052430 con 0x7f35940b1730 2024-03-25T18:39:47.979 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.978+0000 7f3589ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:47.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.978+0000 7f3589ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:47.980 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:39:47.980 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:47.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.980+0000 7f359b8c8640 10 librados: watch_flush enter 2024-03-25T18:39:47.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 10 librados: watch_flush exit 2024-03-25T18:39:47.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 20 client.4221.objecter shutdown clearing up homeless session... 2024-03-25T18:39:47.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 10 client.4221.objecter successfully canceled tick 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 >> v1:172.21.15.136:6800/1141502808 conn(0x7f357806e0b0 legacy=0x7f3578070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 10 monclient: shutdown 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 >> v1:172.21.15.32:6789/0 conn(0x7f35940b1730 legacy=0x7f35941db3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 shutdown_connections 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 >> 172.21.15.32:0/2012258281 conn(0x7f35940617a0 msgr2=0x7f35940b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 shutdown_connections 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 1 -- 172.21.15.32:0/2012258281 wait complete. 2024-03-25T18:39:47.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:47.981+0000 7f359b8c8640 1 librados: shutdown 2024-03-25T18:39:47.992 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:39:50.577 INFO:tasks.ceph.osd.6.smithi136.stderr:2024-03-25T18:39:50.574+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:39:51.002 INFO:tasks.ceph.osd.4.smithi136.stderr:2024-03-25T18:39:50.998+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:39:51.031 INFO:tasks.ceph.osd.7.smithi136.stderr:2024-03-25T18:39:50.953+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:39:51.237 INFO:tasks.ceph.osd.5.smithi136.stderr:2024-03-25T18:39:51.234+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:39:51.571 INFO:tasks.ceph.osd.1.smithi032.stderr:2024-03-25T18:39:51.567+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:39:51.912 INFO:tasks.ceph.osd.3.smithi032.stderr:2024-03-25T18:39:51.907+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:39:52.390 INFO:tasks.ceph.osd.2.smithi032.stderr:2024-03-25T18:39:52.386+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:39:52.801 INFO:tasks.ceph.osd.0.smithi032.stderr:2024-03-25T18:39:52.796+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:39:54.798 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.917+0000 7f23484c8640 10 monclient: get_monmap_and_config 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.917+0000 7f23484c8640 10 monclient: build_initial_monmap 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.917+0000 7f23484c8640 10 monclient: monmap: 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:39:54.918529+0000 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:39:54.918529+0000 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:54.918 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:54.919 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:54.919 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:39:54.919 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:39:54.919 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.917+0000 7f23484c8640 1 Processor -- start 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.917+0000 7f23484c8640 1 -- start start 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.917+0000 7f23484c8640 10 monclient: init 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.918+0000 7f23484c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.918+0000 7f23484c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.918+0000 7f23484c8640 10 monclient(hunting): picked mon.noname-a con 0x7f23400a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.918+0000 7f23484c8640 10 monclient(hunting): picked mon.noname-b con 0x7f234013f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.918+0000 7f23484c8640 10 monclient(hunting): picked mon.noname-c con 0x7f23401430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.918+0000 7f23484c8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2340142520 con 0x7f23400a98f0 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.918+0000 7f23484c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23400b28a0 con 0x7f23401430c0 2024-03-25T18:39:54.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.918+0000 7f23484c8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23400b2a30 con 0x7f234013f710 2024-03-25T18:39:54.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.919+0000 7f23484c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:54.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.919+0000 7f23484c8640 10 monclient(hunting): authenticate will time out at 6101.108583s 2024-03-25T18:39:54.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.921+0000 7f2346a65640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f23401430c0 0x7f234014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:54102/0 (socket says 172.21.15.32:54102) 2024-03-25T18:39:54.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.921+0000 7f2346a65640 1 -- 172.21.15.32:0/2564338878 learned_addr learned my addr 172.21.15.32:0/2564338878 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:54.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.933+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469302981 0 0) 0x7f23400b28a0 con 0x7f23401430c0 2024-03-25T18:39:54.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.933+0000 7f2345262640 10 monclient(hunting): _negotiate 2024-03-25T18:39:54.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.933+0000 7f2345262640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:54.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.933+0000 7f2345262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:54.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.933+0000 7f2345262640 10 monclient(hunting): my global_id is 4243 2024-03-25T18:39:54.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.933+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f232c003520 con 0x7f23401430c0 2024-03-25T18:39:54.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.952+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550618571 0 0) 0x7f2340142520 con 0x7f23400a98f0 2024-03-25T18:39:54.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.952+0000 7f2345262640 10 monclient(hunting): _negotiate 2024-03-25T18:39:54.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.952+0000 7f2345262640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:54.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.952+0000 7f2345262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:54.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.952+0000 7f2345262640 10 monclient(hunting): my global_id is 4248 2024-03-25T18:39:54.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.952+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23400b28a0 con 0x7f23400a98f0 2024-03-25T18:39:54.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.956+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814831637 0 0) 0x7f23400b2a30 con 0x7f234013f710 2024-03-25T18:39:54.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.956+0000 7f2345262640 10 monclient(hunting): _negotiate 2024-03-25T18:39:54.958 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.956+0000 7f2345262640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:54.958 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.956+0000 7f2345262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:54.958 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.956+0000 7f2345262640 10 monclient(hunting): my global_id is 4244 2024-03-25T18:39:54.958 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.956+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2340142520 con 0x7f234013f710 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2569602668 0 0) 0x7f23400b28a0 con 0x7f23400a98f0 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 >> v1:172.21.15.32:6790/0 conn(0x7f234013f710 legacy=0x7f23401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 >> v1:172.21.15.136:6789/0 conn(0x7f23401430c0 legacy=0x7f234014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 10 monclient: _finish_hunting 0 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 1 monclient: found mon.noname-a 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23400b44c0 con 0x7f23400a98f0 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 10 monclient: _finish_auth 0 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 10 monclient: _check_auth_tickets 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f2345262640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f23484c8640 5 monclient: authenticate success, global_id 4248 2024-03-25T18:39:54.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.962+0000 7f23484c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:54.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.966+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f233c001630 con 0x7f23400a98f0 2024-03-25T18:39:54.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.966+0000 7f2345262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.966+0000 7f2345262640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.966+0000 7f2345262640 10 monclient: dump: 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:54.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.966+0000 7f23484c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:39:54.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.979+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f233c001880 con 0x7f23400a98f0 2024-03-25T18:39:54.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.979+0000 7f2345262640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:54.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.979+0000 7f2345262640 1 -- 172.21.15.32:0/2564338878 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f233c001c10 con 0x7f23400a98f0 2024-03-25T18:39:54.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.979+0000 7f23484c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.979+0000 7f23484c8640 4 set_mon_vals no callback set 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.979+0000 7f2345262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.979+0000 7f2345262640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.979+0000 7f2345262640 10 monclient: dump: 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:54.981 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:54.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.980+0000 7f23484c8640 10 monclient: shutdown 2024-03-25T18:39:54.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.980+0000 7f23484c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:54.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.980+0000 7f23484c8640 1 -- 172.21.15.32:0/2564338878 >> v1:172.21.15.32:6789/0 conn(0x7f23400a98f0 legacy=0x7f23400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:54.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.980+0000 7f23484c8640 1 -- 172.21.15.32:0/2564338878 shutdown_connections 2024-03-25T18:39:54.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.980+0000 7f23484c8640 1 -- 172.21.15.32:0/2564338878 >> 172.21.15.32:0/2564338878 conn(0x7f23400b7ea0 msgr2=0x7f23400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:54.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.980+0000 7f23484c8640 1 -- 172.21.15.32:0/2564338878 shutdown_connections 2024-03-25T18:39:54.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.980+0000 7f23484c8640 1 -- 172.21.15.32:0/2564338878 wait complete. 2024-03-25T18:39:54.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.981+0000 7f23484c8640 10 monclient: build_initial_monmap 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.981+0000 7f23484c8640 10 monclient: monmap: 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.982+0000 7f23484c8640 1 librados: starting msgr at 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.982+0000 7f23484c8640 1 librados: starting objecter 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.982+0000 7f23484c8640 1 Processor -- start 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.982+0000 7f23484c8640 1 -- start start 2024-03-25T18:39:54.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.982+0000 7f23484c8640 1 librados: setting wanted keys 2024-03-25T18:39:54.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.982+0000 7f23484c8640 1 librados: calling monclient init 2024-03-25T18:39:54.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.982+0000 7f23484c8640 10 monclient: init 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 10 monclient(hunting): picked mon.noname-b con 0x7f23400a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 10 monclient(hunting): picked mon.noname-a con 0x7f234013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 10 monclient(hunting): picked mon.noname-c con 0x7f23401430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234014e3f0 con 0x7f234013f710 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2340141e30 con 0x7f23400a98f0 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f234013bc30 con 0x7f23401430c0 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:39:54.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.983+0000 7f23484c8640 10 monclient(hunting): authenticate will time out at 6101.172915s 2024-03-25T18:39:54.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.989+0000 7f2345a63640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f234013f710 0x7f234014d5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52018/0 (socket says 172.21.15.32:52018) 2024-03-25T18:39:54.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.989+0000 7f2345a63640 1 -- 172.21.15.32:0/3472973613 learned_addr learned my addr 172.21.15.32:0/3472973613 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:39:54.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.990+0000 7f2336ffd640 10 client.?.objecter ms_handle_connect 0x7f234013f710 2024-03-25T18:39:54.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.990+0000 7f2336ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:54.995 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.994+0000 7f2336ffd640 10 client.?.objecter ms_handle_connect 0x7f23401430c0 2024-03-25T18:39:54.995 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:54.994+0000 7f2336ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:55.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.000+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460443513 0 0) 0x7f234013bc30 con 0x7f23401430c0 2024-03-25T18:39:55.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.000+0000 7f2336ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:55.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.000+0000 7f2336ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:55.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.000+0000 7f2336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:55.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.000+0000 7f2336ffd640 10 monclient(hunting): my global_id is 4247 2024-03-25T18:39:55.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.000+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2324004eb0 con 0x7f23401430c0 2024-03-25T18:39:55.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.003+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3654513294 0 0) 0x7f234014e3f0 con 0x7f234013f710 2024-03-25T18:39:55.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.003+0000 7f2336ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:55.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.003+0000 7f2336ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:55.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.003+0000 7f2336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:55.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.003+0000 7f2336ffd640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:39:55.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.003+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234013bc30 con 0x7f234013f710 2024-03-25T18:39:55.007 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.006+0000 7f2336ffd640 10 client.?.objecter ms_handle_connect 0x7f23400a98f0 2024-03-25T18:39:55.007 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.006+0000 7f2336ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:39:55.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.012+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287583180 0 0) 0x7f2340141e30 con 0x7f23400a98f0 2024-03-25T18:39:55.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.013+0000 7f2336ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:39:55.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.013+0000 7f2336ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:39:55.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.013+0000 7f2336ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:39:55.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.013+0000 7f2336ffd640 10 monclient(hunting): my global_id is 4246 2024-03-25T18:39:55.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.013+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f234014e3f0 con 0x7f23400a98f0 2024-03-25T18:39:55.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.018+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 529405130 0 0) 0x7f2324004eb0 con 0x7f23401430c0 2024-03-25T18:39:55.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.018+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2340141e30 con 0x7f23401430c0 2024-03-25T18:39:55.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.018+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2338001af0 con 0x7f23401430c0 2024-03-25T18:39:55.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.018+0000 7f2336ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:55.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.018+0000 7f2336ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:39:55.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.018+0000 7f2336ffd640 10 monclient(hunting): dump: 2024-03-25T18:39:55.019 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:55.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.018+0000 7f2336ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.025+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 751809209 0 0) 0x7f234014e3f0 con 0x7f23400a98f0 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.025+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2324004eb0 con 0x7f23400a98f0 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.025+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f233c001b90 con 0x7f23400a98f0 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.025+0000 7f2336ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.025+0000 7f2336ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.025+0000 7f2336ffd640 10 monclient(hunting): dump: 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:55.026 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:55.027 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:55.027 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:55.027 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:55.027 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:55.027 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:55.027 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3207824889 0 0) 0x7f2340141e30 con 0x7f23401430c0 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 >> v1:172.21.15.136:6789/0 conn(0x7f23400a98f0 legacy=0x7f234013b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 >> v1:172.21.15.32:6789/0 conn(0x7f234013f710 legacy=0x7f234014d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 1 monclient: found mon.c 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23400b6c20 con 0x7f23401430c0 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 10 monclient: _finish_auth 0 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f2336ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 5 monclient: authenticate success, global_id 4247 2024-03-25T18:39:55.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 10 monclient: _renew_subs 2024-03-25T18:39:55.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:55.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23400b5a90 con 0x7f23401430c0 2024-03-25T18:39:55.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:39:55.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 10 monclient: _renew_subs 2024-03-25T18:39:55.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:55.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2340139140 con 0x7f23401430c0 2024-03-25T18:39:55.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.027+0000 7f23484c8640 1 librados: init done 2024-03-25T18:39:55.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.028+0000 7f23484c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:55.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.028+0000 7f23484c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:55.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.028+0000 7f23484c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:55.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.028+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23400b3ef0 con 0x7f23401430c0 2024-03-25T18:39:55.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.248+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2338001f20 con 0x7f23401430c0 2024-03-25T18:39:55.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.248+0000 7f2336ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:39:55.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.248+0000 7f2344a61640 4 set_mon_vals no callback set 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.338+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2338004d00 con 0x7f23401430c0 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.338+0000 7f2336ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.338+0000 7f2336ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.338+0000 7f2336ffd640 10 monclient: dump: 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:39:55.339 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:39:55.340 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:39:55.340 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:39:55.340 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:39:55.340 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:39:55.340 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.338+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f2338004ee0 con 0x7f23401430c0 2024-03-25T18:39:55.340 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.339+0000 7f2346264640 10 monclient: get_auth_request con 0x7f232406df90 auth_method 0 2024-03-25T18:39:55.340 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.339+0000 7f2336ffd640 10 client.4247.objecter ms_handle_connect 0x7f232406df90 2024-03-25T18:39:55.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.344+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f2338054980 con 0x7f23401430c0 2024-03-25T18:39:55.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.344+0000 7f2336ffd640 10 client.4247.objecter ms_dispatch 0x7f2340000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:39:55.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.344+0000 7f2336ffd640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:39:55.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.344+0000 7f2336ffd640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:39:55.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.344+0000 7f2336ffd640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-25T18:39:55.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.344+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2338050490 con 0x7f23401430c0 2024-03-25T18:39:55.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.344+0000 7f2336ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:39:55.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.344+0000 7f2336ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:39:55.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.605+0000 7f23484c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:55.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.605+0000 7f23484c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:55.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.605+0000 7f23484c8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:39:55.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.605+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f23400b6a20 con 0x7f23401430c0 2024-03-25T18:39:55.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.638+0000 7f2336ffd640 1 -- 172.21.15.32:0/3472973613 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f23380523c0 con 0x7f23401430c0 2024-03-25T18:39:55.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.638+0000 7f2336ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:39:55.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.638+0000 7f2336ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:39:55.640 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:39:55.640 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:39:55.642 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.640+0000 7f23484c8640 10 librados: watch_flush enter 2024-03-25T18:39:55.642 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.640+0000 7f23484c8640 10 librados: watch_flush exit 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.640+0000 7f23484c8640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.640+0000 7f23484c8640 10 client.4247.objecter successfully canceled tick 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 >> v1:172.21.15.136:6800/1141502808 conn(0x7f232406df90 legacy=0x7f2324070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 10 monclient: shutdown 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 >> v1:172.21.15.32:6790/0 conn(0x7f23401430c0 legacy=0x7f234014dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 shutdown_connections 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 >> 172.21.15.32:0/3472973613 conn(0x7f23400b7ea0 msgr2=0x7f234013d2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 shutdown_connections 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 1 -- 172.21.15.32:0/3472973613 wait complete. 2024-03-25T18:39:55.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:39:55.641+0000 7f23484c8640 1 librados: shutdown 2024-03-25T18:39:55.653 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:40:02.460 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:40:02.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.571+0000 7f705d583640 10 monclient: get_monmap_and_config 2024-03-25T18:40:02.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.571+0000 7f705d583640 10 monclient: build_initial_monmap 2024-03-25T18:40:02.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.571+0000 7f705d583640 10 monclient: monmap: 2024-03-25T18:40:02.572 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:02.572 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:40:02.572376+0000 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:40:02.572376+0000 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.572+0000 7f705d583640 1 Processor -- start 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.572+0000 7f705d583640 1 -- start start 2024-03-25T18:40:02.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.572+0000 7f705d583640 10 monclient: init 2024-03-25T18:40:02.574 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 10 monclient(hunting): picked mon.noname-b con 0x7f70580a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 10 monclient(hunting): picked mon.noname-a con 0x7f705813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 10 monclient(hunting): picked mon.noname-c con 0x7f70581430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058142520 con 0x7f705813f710 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580b28a0 con 0x7f70581430c0 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70580b2a30 con 0x7f70580a98f0 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:02.575 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.573+0000 7f705d583640 10 monclient(hunting): authenticate will time out at 6108.762067s 2024-03-25T18:40:02.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.576+0000 7f7056ffd640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f70580a98f0 0x7f70580a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:35894/0 (socket says 172.21.15.32:35894) 2024-03-25T18:40:02.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.576+0000 7f7056ffd640 1 -- 172.21.15.32:0/1096096583 learned_addr learned my addr 172.21.15.32:0/1096096583 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:02.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.596+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3547139860 0 0) 0x7f70580b28a0 con 0x7f70581430c0 2024-03-25T18:40:02.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.596+0000 7f7055ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:40:02.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.596+0000 7f7055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:02.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.596+0000 7f7055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:02.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.596+0000 7f7055ffb640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:40:02.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.596+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7040003520 con 0x7f70581430c0 2024-03-25T18:40:02.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.601+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 252537279 0 0) 0x7f7058142520 con 0x7f705813f710 2024-03-25T18:40:02.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.601+0000 7f7055ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:40:02.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.601+0000 7f7055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:02.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.601+0000 7f7055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:02.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.601+0000 7f7055ffb640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:40:02.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.601+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f70580b28a0 con 0x7f705813f710 2024-03-25T18:40:02.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.610+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2766516848 0 0) 0x7f70580b2a30 con 0x7f70580a98f0 2024-03-25T18:40:02.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.610+0000 7f7055ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:40:02.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.610+0000 7f7055ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:02.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.610+0000 7f7055ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:02.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.610+0000 7f7055ffb640 10 monclient(hunting): my global_id is 4250 2024-03-25T18:40:02.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.610+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7058142520 con 0x7f70580a98f0 2024-03-25T18:40:02.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 94994042 0 0) 0x7f7040003520 con 0x7f70581430c0 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 >> v1:172.21.15.32:6790/0 conn(0x7f70580a98f0 legacy=0x7f70580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 >> v1:172.21.15.32:6789/0 conn(0x7f705813f710 legacy=0x7f70581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 1 monclient: found mon.noname-c 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70580b44c0 con 0x7f70581430c0 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 10 monclient: _finish_auth 0 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:02.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7050001a10 con 0x7f70581430c0 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f7055ffb640 10 monclient: dump: 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f705d583640 5 monclient: authenticate success, global_id 4249 2024-03-25T18:40:02.613 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.611+0000 7f705d583640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:02.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.626+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7050001b70 con 0x7f70581430c0 2024-03-25T18:40:02.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.626+0000 7f7055ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:02.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.626+0000 7f7055ffb640 1 -- 172.21.15.32:0/1096096583 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7050001f80 con 0x7f70581430c0 2024-03-25T18:40:02.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.626+0000 7f7055ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:02.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.626+0000 7f7055ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.626+0000 7f7055ffb640 10 monclient: dump: 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.626+0000 7f705d583640 10 monclient: get_monmap_and_config success 2024-03-25T18:40:02.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.626+0000 7f705d583640 4 set_mon_vals no callback set 2024-03-25T18:40:02.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 10 monclient: shutdown 2024-03-25T18:40:02.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:02.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 1 -- 172.21.15.32:0/1096096583 >> v1:172.21.15.136:6789/0 conn(0x7f70581430c0 legacy=0x7f705814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:02.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 1 -- 172.21.15.32:0/1096096583 shutdown_connections 2024-03-25T18:40:02.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 1 -- 172.21.15.32:0/1096096583 >> 172.21.15.32:0/1096096583 conn(0x7f70580b7ea0 msgr2=0x7f70580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:02.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 1 -- 172.21.15.32:0/1096096583 shutdown_connections 2024-03-25T18:40:02.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 1 -- 172.21.15.32:0/1096096583 wait complete. 2024-03-25T18:40:02.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 10 monclient: build_initial_monmap 2024-03-25T18:40:02.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.627+0000 7f705d583640 10 monclient: monmap: 2024-03-25T18:40:02.632 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:02.632 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:40:02.632 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:40:02.632 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:40:02.632 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:02.632 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:40:02.632 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 1 librados: starting msgr at 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 1 librados: starting objecter 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 1 Processor -- start 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 1 -- start start 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 1 librados: setting wanted keys 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 1 librados: calling monclient init 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 10 monclient: init 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:40:02.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 10 monclient(hunting): picked mon.noname-c con 0x7f70580a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:02.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.628+0000 7f705d583640 10 monclient(hunting): picked mon.noname-a con 0x7f705813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:02.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.629+0000 7f705d583640 10 monclient(hunting): picked mon.noname-b con 0x7f70581430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:02.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.629+0000 7f705d583640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058136ec0 con 0x7f705813f710 2024-03-25T18:40:02.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.629+0000 7f705d583640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058136c20 con 0x7f70581430c0 2024-03-25T18:40:02.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.629+0000 7f705d583640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7058137db0 con 0x7f70580a98f0 2024-03-25T18:40:02.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.629+0000 7f705d583640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:02.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.629+0000 7f705d583640 10 monclient(hunting): authenticate will time out at 6108.817577s 2024-03-25T18:40:02.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.633+0000 7f70577fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f70581430c0 0x7f70581e2c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:36270/0 (socket says 172.21.15.32:36270) 2024-03-25T18:40:02.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.633+0000 7f70577fe640 1 -- 172.21.15.32:0/2573838090 learned_addr learned my addr 172.21.15.32:0/2573838090 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:02.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.634+0000 7f703f7fe640 10 client.?.objecter ms_handle_connect 0x7f70581430c0 2024-03-25T18:40:02.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.634+0000 7f703f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:02.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.641+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3461670296 0 0) 0x7f7058136c20 con 0x7f70581430c0 2024-03-25T18:40:02.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.641+0000 7f703f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:02.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.641+0000 7f703f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:02.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.641+0000 7f703f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:02.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.641+0000 7f703f7fe640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:40:02.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.641+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7044004eb0 con 0x7f70581430c0 2024-03-25T18:40:02.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.649+0000 7f703f7fe640 10 client.?.objecter ms_handle_connect 0x7f705813f710 2024-03-25T18:40:02.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.649+0000 7f703f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:02.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.653+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919632753 0 0) 0x7f7044004eb0 con 0x7f70581430c0 2024-03-25T18:40:02.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.653+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7058136c20 con 0x7f70581430c0 2024-03-25T18:40:02.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.653+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7050001f40 con 0x7f70581430c0 2024-03-25T18:40:02.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.653+0000 7f703f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:02.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.653+0000 7f703f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:40:02.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.653+0000 7f703f7fe640 10 monclient(hunting): dump: 2024-03-25T18:40:02.653 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:02.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.653+0000 7f703f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:40:02.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.658+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617200117 0 0) 0x7f7058136ec0 con 0x7f705813f710 2024-03-25T18:40:02.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.658+0000 7f703f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:02.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.658+0000 7f703f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:02.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.658+0000 7f703f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:02.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.658+0000 7f703f7fe640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:40:02.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.658+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7044004eb0 con 0x7f705813f710 2024-03-25T18:40:02.661 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.661+0000 7f703f7fe640 10 client.?.objecter ms_handle_connect 0x7f70580a98f0 2024-03-25T18:40:02.661 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.661+0000 7f703f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:02.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 41772896 0 0) 0x7f7058136c20 con 0x7f70581430c0 2024-03-25T18:40:02.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 >> v1:172.21.15.32:6790/0 conn(0x7f70580a98f0 legacy=0x7f70581376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:02.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 >> v1:172.21.15.32:6789/0 conn(0x7f705813f710 legacy=0x7f70580b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:02.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:40:02.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 1 monclient: found mon.b 2024-03-25T18:40:02.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7058138b50 con 0x7f70581430c0 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f703f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 10 monclient: _renew_subs 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7058139830 con 0x7f70581430c0 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 10 monclient: _renew_subs 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:02.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f70581369b0 con 0x7f70581430c0 2024-03-25T18:40:02.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.664+0000 7f705d583640 1 librados: init done 2024-03-25T18:40:02.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.665+0000 7f705d583640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:02.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.665+0000 7f705d583640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:02.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.665+0000 7f705d583640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:02.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.665+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70580627b0 con 0x7f70581430c0 2024-03-25T18:40:02.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.935+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7050001510 con 0x7f70581430c0 2024-03-25T18:40:02.937 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.936+0000 7f703f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:02.937 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.936+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7050003dc0 con 0x7f70581430c0 2024-03-25T18:40:02.937 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.936+0000 7f703f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:02.937 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.936+0000 7f703f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:40:02.937 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.936+0000 7f703f7fe640 10 monclient: dump: 2024-03-25T18:40:02.937 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.936+0000 7f70557fa640 4 set_mon_vals no callback set 2024-03-25T18:40:02.938 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.936+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f7050003fe0 con 0x7f70581430c0 2024-03-25T18:40:02.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.937+0000 7f7056ffd640 10 monclient: get_auth_request con 0x7f704406dd40 auth_method 0 2024-03-25T18:40:02.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.937+0000 7f703f7fe640 10 client.4252.objecter ms_handle_connect 0x7f704406dd40 2024-03-25T18:40:02.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.942+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f7050054ba0 con 0x7f70581430c0 2024-03-25T18:40:02.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.942+0000 7f703f7fe640 10 client.4252.objecter ms_dispatch 0x7f7058000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:40:02.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.942+0000 7f703f7fe640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:40:02.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.942+0000 7f703f7fe640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:40:02.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.942+0000 7f703f7fe640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T18:40:02.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.942+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f70500504e0 con 0x7f70581430c0 2024-03-25T18:40:02.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.942+0000 7f703f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:02.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:02.942+0000 7f703f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:40:03.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.198+0000 7f705d583640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:03.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.198+0000 7f705d583640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:03.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.198+0000 7f705d583640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:03.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.198+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7058069f40 con 0x7f70581430c0 2024-03-25T18:40:03.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.215+0000 7f703f7fe640 1 -- 172.21.15.32:0/2573838090 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f70500525e0 con 0x7f70581430c0 2024-03-25T18:40:03.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.215+0000 7f703f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:03.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.215+0000 7f703f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:40:03.216 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:40:03.216 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:40:03.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 10 librados: watch_flush enter 2024-03-25T18:40:03.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 10 librados: watch_flush exit 2024-03-25T18:40:03.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T18:40:03.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 10 client.4252.objecter successfully canceled tick 2024-03-25T18:40:03.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 >> v1:172.21.15.136:6800/1141502808 conn(0x7f704406dd40 legacy=0x7f7044070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:03.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 10 monclient: shutdown 2024-03-25T18:40:03.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:03.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 >> v1:172.21.15.136:6789/0 conn(0x7f70581430c0 legacy=0x7f70581e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:03.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.217+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 shutdown_connections 2024-03-25T18:40:03.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.218+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 >> 172.21.15.32:0/2573838090 conn(0x7f70580b7ea0 msgr2=0x7f705813ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:03.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.218+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 shutdown_connections 2024-03-25T18:40:03.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.218+0000 7f705d583640 1 -- 172.21.15.32:0/2573838090 wait complete. 2024-03-25T18:40:03.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:03.218+0000 7f705d583640 1 librados: shutdown 2024-03-25T18:40:03.230 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:40:07.132 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:19.459 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:07.194 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:19.522 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:07.366 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:19.682 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:07.371 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:19.699 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:07.424 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:19.741 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:07.441 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:19.769 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:07.459 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:19.787 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:07.638 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:19.955 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:07.673 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:20.000 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:10.037 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:40:10.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.151+0000 7f292fa6b640 10 monclient: get_monmap_and_config 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.151+0000 7f292fa6b640 10 monclient: build_initial_monmap 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.151+0000 7f292fa6b640 10 monclient: monmap: 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:40:10.152029+0000 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:40:10.152029+0000 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:40:10.152 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:10.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.152+0000 7f292fa6b640 1 Processor -- start 2024-03-25T18:40:10.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.152+0000 7f292fa6b640 1 -- start start 2024-03-25T18:40:10.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.152+0000 7f292fa6b640 10 monclient: init 2024-03-25T18:40:10.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:10.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:40:10.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 10 monclient(hunting): picked mon.noname-a con 0x7f29280a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:10.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 10 monclient(hunting): picked mon.noname-b con 0x7f292813f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:10.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 10 monclient(hunting): picked mon.noname-c con 0x7f29281430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:10.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2928142520 con 0x7f29280a98f0 2024-03-25T18:40:10.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280b28a0 con 0x7f29281430c0 2024-03-25T18:40:10.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29280b2a30 con 0x7f292813f710 2024-03-25T18:40:10.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:10.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.153+0000 7f292fa6b640 10 monclient(hunting): authenticate will time out at 6116.342374s 2024-03-25T18:40:10.165 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.166+0000 7f292e008640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f29281430c0 0x7f292814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:38272/0 (socket says 172.21.15.32:38272) 2024-03-25T18:40:10.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.166+0000 7f292e008640 1 -- 172.21.15.32:0/4283964732 learned_addr learned my addr 172.21.15.32:0/4283964732 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:10.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.180+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1281998282 0 0) 0x7f29280b28a0 con 0x7f29281430c0 2024-03-25T18:40:10.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.180+0000 7f292c805640 10 monclient(hunting): _negotiate 2024-03-25T18:40:10.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.180+0000 7f292c805640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:10.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.180+0000 7f292c805640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:10.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.180+0000 7f292c805640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:40:10.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.180+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2914003520 con 0x7f29281430c0 2024-03-25T18:40:10.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.188+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3559259914 0 0) 0x7f2928142520 con 0x7f29280a98f0 2024-03-25T18:40:10.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.188+0000 7f292c805640 10 monclient(hunting): _negotiate 2024-03-25T18:40:10.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.188+0000 7f292c805640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:10.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.188+0000 7f292c805640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:10.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.188+0000 7f292c805640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:40:10.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.188+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29280b28a0 con 0x7f29280a98f0 2024-03-25T18:40:10.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.193+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2477105202 0 0) 0x7f29280b2a30 con 0x7f292813f710 2024-03-25T18:40:10.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.193+0000 7f292c805640 10 monclient(hunting): _negotiate 2024-03-25T18:40:10.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.193+0000 7f292c805640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:10.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.193+0000 7f292c805640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:10.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.193+0000 7f292c805640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:40:10.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.193+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2928142520 con 0x7f292813f710 2024-03-25T18:40:10.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 873124876 0 0) 0x7f29280b28a0 con 0x7f29280a98f0 2024-03-25T18:40:10.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 >> v1:172.21.15.32:6790/0 conn(0x7f292813f710 legacy=0x7f29281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:10.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 >> v1:172.21.15.136:6789/0 conn(0x7f29281430c0 legacy=0x7f292814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:10.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 10 monclient: _finish_hunting 0 2024-03-25T18:40:10.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 1 monclient: found mon.noname-a 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280b44c0 con 0x7f29280a98f0 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 10 monclient: _finish_auth 0 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 10 monclient: _check_auth_tickets 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2924001670 con 0x7f29280a98f0 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292c805640 10 monclient: dump: 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:10.202 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:10.203 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:10.203 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:10.203 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:10.203 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:10.203 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:10.203 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:10.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292fa6b640 5 monclient: authenticate success, global_id 4260 2024-03-25T18:40:10.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.201+0000 7f292fa6b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.222+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29240017d0 con 0x7f29280a98f0 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292c805640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292c805640 1 -- 172.21.15.32:0/4283964732 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2924001be0 con 0x7f29280a98f0 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 10 monclient: get_monmap_and_config success 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 4 set_mon_vals no callback set 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292c805640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292c805640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292c805640 10 monclient: dump: 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:10.223 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:10.224 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:10.224 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:10.224 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:10.224 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:10.224 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:10.224 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:10.226 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 10 monclient: shutdown 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 1 -- 172.21.15.32:0/4283964732 >> v1:172.21.15.32:6789/0 conn(0x7f29280a98f0 legacy=0x7f29280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 1 -- 172.21.15.32:0/4283964732 shutdown_connections 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 1 -- 172.21.15.32:0/4283964732 >> 172.21.15.32:0/4283964732 conn(0x7f29280b7ea0 msgr2=0x7f29280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 1 -- 172.21.15.32:0/4283964732 shutdown_connections 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.223+0000 7f292fa6b640 1 -- 172.21.15.32:0/4283964732 wait complete. 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.224+0000 7f292fa6b640 10 monclient: build_initial_monmap 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.224+0000 7f292fa6b640 10 monclient: monmap: 2024-03-25T18:40:10.227 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:10.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 librados: starting msgr at 2024-03-25T18:40:10.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 librados: starting objecter 2024-03-25T18:40:10.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 Processor -- start 2024-03-25T18:40:10.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 -- start start 2024-03-25T18:40:10.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 librados: setting wanted keys 2024-03-25T18:40:10.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 librados: calling monclient init 2024-03-25T18:40:10.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 10 monclient: init 2024-03-25T18:40:10.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:10.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:40:10.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 10 monclient(hunting): picked mon.noname-b con 0x7f29280a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:10.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 10 monclient(hunting): picked mon.noname-a con 0x7f292813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:10.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 10 monclient(hunting): picked mon.noname-c con 0x7f29281430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:10.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2928137030 con 0x7f292813f710 2024-03-25T18:40:10.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29281398a0 con 0x7f29280a98f0 2024-03-25T18:40:10.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29281d4c90 con 0x7f29281430c0 2024-03-25T18:40:10.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:10.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.225+0000 7f292fa6b640 10 monclient(hunting): authenticate will time out at 6116.414484s 2024-03-25T18:40:10.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.230+0000 7f292d006640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f292813f710 0x7f29280aff00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36440/0 (socket says 172.21.15.32:36440) 2024-03-25T18:40:10.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.230+0000 7f292d006640 1 -- 172.21.15.32:0/1229644589 learned_addr learned my addr 172.21.15.32:0/1229644589 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:10.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.234+0000 7f291e7fc640 10 client.?.objecter ms_handle_connect 0x7f292813f710 2024-03-25T18:40:10.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.234+0000 7f291e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:10.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.237+0000 7f291e7fc640 10 client.?.objecter ms_handle_connect 0x7f29281430c0 2024-03-25T18:40:10.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.237+0000 7f291e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:10.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.244+0000 7f291e7fc640 10 client.?.objecter ms_handle_connect 0x7f29280a98f0 2024-03-25T18:40:10.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.244+0000 7f291e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:10.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.244+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3005535330 0 0) 0x7f2928137030 con 0x7f292813f710 2024-03-25T18:40:10.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.244+0000 7f291e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:40:10.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.244+0000 7f291e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:10.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.244+0000 7f291e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:10.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.244+0000 7f291e7fc640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:40:10.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.244+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f290c004eb0 con 0x7f292813f710 2024-03-25T18:40:10.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.258+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 865329646 0 0) 0x7f290c004eb0 con 0x7f292813f710 2024-03-25T18:40:10.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.259+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2928137030 con 0x7f292813f710 2024-03-25T18:40:10.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.259+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2918001a10 con 0x7f292813f710 2024-03-25T18:40:10.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.259+0000 7f291e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:10.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.259+0000 7f291e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:40:10.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.259+0000 7f291e7fc640 10 monclient(hunting): dump: 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:10.260 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:10.261 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:10.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.259+0000 7f291e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:40:10.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1124393719 0 0) 0x7f2928137030 con 0x7f292813f710 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 >> v1:172.21.15.32:6790/0 conn(0x7f29281430c0 legacy=0x7f29281d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 >> v1:172.21.15.136:6789/0 conn(0x7f29280a98f0 legacy=0x7f29281d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 1 monclient: found mon.a 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29280b16b0 con 0x7f292813f710 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f291e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:10.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 5 monclient: authenticate success, global_id 4263 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 10 monclient: _renew_subs 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29280b0c10 con 0x7f292813f710 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 10 client.4263.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 10 monclient: _renew_subs 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29280b07f0 con 0x7f292813f710 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.329+0000 7f292fa6b640 1 librados: init done 2024-03-25T18:40:10.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.330+0000 7f292fa6b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:10.332 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.330+0000 7f292fa6b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:10.332 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.330+0000 7f292fa6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:10.332 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.330+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29281d4e20 con 0x7f292813f710 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.646+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2918001cb0 con 0x7f292813f710 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.646+0000 7f291e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.646+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2918003a60 con 0x7f292813f710 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.646+0000 7f291e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.646+0000 7f291e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.646+0000 7f291e7fc640 10 monclient: dump: 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:10.647 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:10.648 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:10.648 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:10.648 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:10.648 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:10.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.646+0000 7f291ffff640 4 set_mon_vals no callback set 2024-03-25T18:40:10.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.647+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f2918003c80 con 0x7f292813f710 2024-03-25T18:40:10.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.647+0000 7f292d807640 10 monclient: get_auth_request con 0x7f290c06dd00 auth_method 0 2024-03-25T18:40:10.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.648+0000 7f291e7fc640 10 client.4263.objecter ms_handle_connect 0x7f290c06dd00 2024-03-25T18:40:10.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.652+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f2918054890 con 0x7f292813f710 2024-03-25T18:40:10.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.652+0000 7f291e7fc640 10 client.4263.objecter ms_dispatch 0x7f2928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:40:10.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.652+0000 7f291e7fc640 3 client.4263.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:40:10.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.652+0000 7f291e7fc640 3 client.4263.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:40:10.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.652+0000 7f291e7fc640 20 client.4263.objecter dump_active .. 0 homeless 2024-03-25T18:40:10.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.652+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29180503a0 con 0x7f292813f710 2024-03-25T18:40:10.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.652+0000 7f291e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:10.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.652+0000 7f291e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:40:10.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.867+0000 7f292fa6b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:10.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.867+0000 7f292fa6b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:10.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.867+0000 7f292fa6b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:10.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.867+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29280627b0 con 0x7f292813f710 2024-03-25T18:40:10.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.897+0000 7f291e7fc640 1 -- 172.21.15.32:0/1229644589 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f29180522d0 con 0x7f292813f710 2024-03-25T18:40:10.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.897+0000 7f291e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:10.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.897+0000 7f291e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:40:10.897 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:40:10.897 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 10 librados: watch_flush enter 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 10 librados: watch_flush exit 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 20 client.4263.objecter shutdown clearing up homeless session... 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 10 client.4263.objecter successfully canceled tick 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 >> v1:172.21.15.136:6800/1141502808 conn(0x7f290c06dd00 legacy=0x7f290c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 10 monclient: shutdown 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 >> v1:172.21.15.32:6789/0 conn(0x7f292813f710 legacy=0x7f29280aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 shutdown_connections 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 >> 172.21.15.32:0/1229644589 conn(0x7f29280b7ea0 msgr2=0x7f2928074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:10.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 shutdown_connections 2024-03-25T18:40:10.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 1 -- 172.21.15.32:0/1229644589 wait complete. 2024-03-25T18:40:10.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:10.899+0000 7f292fa6b640 1 librados: shutdown 2024-03-25T18:40:10.911 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:40:17.718 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:40:17.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.845+0000 7f1c3a723640 10 monclient: get_monmap_and_config 2024-03-25T18:40:17.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.845+0000 7f1c3a723640 10 monclient: build_initial_monmap 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.845+0000 7f1c3a723640 10 monclient: monmap: 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:40:17.845902+0000 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:40:17.845902+0000 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:40:17.846 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:17.847 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 1 Processor -- start 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 1 -- start start 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 10 monclient: init 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-c con 0x7f1c340ae6e0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-a con 0x7f1c3406f080 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-b con 0x7f1c340704e0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340718d0 con 0x7f1c3406f080 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c3406fa30 con 0x7f1c340ae6e0 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c34070090 con 0x7f1c340704e0 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:17.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.846+0000 7f1c3a723640 10 monclient(hunting): authenticate will time out at 6124.035467s 2024-03-25T18:40:17.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.867+0000 7f1c33fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1c340ae6e0 0x7f1c3413c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:38586/0 (socket says 172.21.15.32:38586) 2024-03-25T18:40:17.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.867+0000 7f1c33fff640 1 -- 172.21.15.32:0/368242527 learned_addr learned my addr 172.21.15.32:0/368242527 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:17.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.877+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3384328595 0 0) 0x7f1c3406fa30 con 0x7f1c340ae6e0 2024-03-25T18:40:17.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.877+0000 7f1c32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:40:17.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.877+0000 7f1c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:17.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.877+0000 7f1c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:17.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.877+0000 7f1c32ffd640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:40:17.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.877+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c1c003520 con 0x7f1c340ae6e0 2024-03-25T18:40:17.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.885+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4279848721 0 0) 0x7f1c34070090 con 0x7f1c340704e0 2024-03-25T18:40:17.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.885+0000 7f1c32ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:40:17.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.885+0000 7f1c32ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:17.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.885+0000 7f1c32ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:17.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.885+0000 7f1c32ffd640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:40:17.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.885+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c3406fa30 con 0x7f1c340704e0 2024-03-25T18:40:17.893 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 209821891 0 0) 0x7f1c1c003520 con 0x7f1c340ae6e0 2024-03-25T18:40:17.893 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 >> v1:172.21.15.32:6790/0 conn(0x7f1c340704e0 legacy=0x7f1c3414a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:17.893 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 >> v1:172.21.15.32:6789/0 conn(0x7f1c3406f080 legacy=0x7f1c34071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:17.893 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:40:17.893 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 1 monclient: found mon.noname-c 2024-03-25T18:40:17.893 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c3406f510 con 0x7f1c340ae6e0 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 10 monclient: _finish_auth 0 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c24001690 con 0x7f1c340ae6e0 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c32ffd640 10 monclient: dump: 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.894 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:17.895 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:17.895 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:17.895 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:17.895 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:17.895 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:17.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c3a723640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:40:17.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.893+0000 7f1c3a723640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:17.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.908+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c240017f0 con 0x7f1c340ae6e0 2024-03-25T18:40:17.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.908+0000 7f1c32ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:17.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.908+0000 7f1c32ffd640 1 -- 172.21.15.32:0/368242527 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c24001c00 con 0x7f1c340ae6e0 2024-03-25T18:40:17.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.908+0000 7f1c3a723640 10 monclient: get_monmap_and_config success 2024-03-25T18:40:17.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.908+0000 7f1c3a723640 4 set_mon_vals no callback set 2024-03-25T18:40:17.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.908+0000 7f1c32ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:17.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.908+0000 7f1c32ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:40:17.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.908+0000 7f1c32ffd640 10 monclient: dump: 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:17.909 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 10 monclient: shutdown 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 1 -- 172.21.15.32:0/368242527 >> v1:172.21.15.136:6789/0 conn(0x7f1c340ae6e0 legacy=0x7f1c3413c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 1 -- 172.21.15.32:0/368242527 shutdown_connections 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 1 -- 172.21.15.32:0/368242527 >> 172.21.15.32:0/368242527 conn(0x7f1c3406b630 msgr2=0x7f1c340b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 1 -- 172.21.15.32:0/368242527 shutdown_connections 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 1 -- 172.21.15.32:0/368242527 wait complete. 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 10 monclient: build_initial_monmap 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.909+0000 7f1c3a723640 10 monclient: monmap: 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:40:17.912 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.910+0000 7f1c3a723640 1 librados: starting msgr at 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.910+0000 7f1c3a723640 1 librados: starting objecter 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.910+0000 7f1c3a723640 1 Processor -- start 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.910+0000 7f1c3a723640 1 -- start start 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.910+0000 7f1c3a723640 1 librados: setting wanted keys 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.910+0000 7f1c3a723640 1 librados: calling monclient init 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.910+0000 7f1c3a723640 10 monclient: init 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:40:17.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-a con 0x7f1c340704e0 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:17.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-b con 0x7f1c340ae6e0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:17.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 10 monclient(hunting): picked mon.noname-c con 0x7f1c341d5a30 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:17.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340abb60 con 0x7f1c340704e0 2024-03-25T18:40:17.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c341d2e60 con 0x7f1c340ae6e0 2024-03-25T18:40:17.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c340ac080 con 0x7f1c341d5a30 2024-03-25T18:40:17.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:17.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.911+0000 7f1c3a723640 10 monclient(hunting): authenticate will time out at 6124.100072s 2024-03-25T18:40:17.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.929+0000 7f1c337fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1c340ae6e0 0x7f1c34136c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:38628/0 (socket says 172.21.15.32:38628) 2024-03-25T18:40:17.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.929+0000 7f1c337fe640 1 -- 172.21.15.32:0/748880835 learned_addr learned my addr 172.21.15.32:0/748880835 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:17.935 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.935+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c340ae6e0 2024-03-25T18:40:17.935 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.935+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:17.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.939+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c341d5a30 2024-03-25T18:40:17.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.939+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:17.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.942+0000 7f1c30ff9640 10 client.?.objecter ms_handle_connect 0x7f1c340704e0 2024-03-25T18:40:17.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.942+0000 7f1c30ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:17.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.943+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2609467887 0 0) 0x7f1c341d2e60 con 0x7f1c340ae6e0 2024-03-25T18:40:17.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.943+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:40:17.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.943+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:17.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.943+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:17.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.943+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4261 2024-03-25T18:40:17.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.943+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c20004eb0 con 0x7f1c340ae6e0 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.956+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547279997 0 0) 0x7f1c340ac080 con 0x7f1c341d5a30 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c341d2e60 con 0x7f1c341d5a30 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1788924813 0 0) 0x7f1c340abb60 con 0x7f1c340704e0 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:17.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:40:17.958 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.957+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c340ac080 con 0x7f1c340704e0 2024-03-25T18:40:17.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.962+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 154655386 0 0) 0x7f1c20004eb0 con 0x7f1c340ae6e0 2024-03-25T18:40:17.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.962+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c340abb60 con 0x7f1c340ae6e0 2024-03-25T18:40:17.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.962+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c28001a00 con 0x7f1c340ae6e0 2024-03-25T18:40:17.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.962+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:17.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.962+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:40:17.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.962+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:40:17.962 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:17.962 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:17.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.962+0000 7f1c30ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:40:17.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.967+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1944912864 0 0) 0x7f1c340ac080 con 0x7f1c340704e0 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.967+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c20004eb0 con 0x7f1c340704e0 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.967+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c24001b00 con 0x7f1c340704e0 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.967+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.967+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.967+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:17.968 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:17.969 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.969+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1388875861 0 0) 0x7f1c341d2e60 con 0x7f1c341d5a30 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.969+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c340ac080 con 0x7f1c341d5a30 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.969+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c2c001cb0 con 0x7f1c341d5a30 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.969+0000 7f1c30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.969+0000 7f1c30ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.969+0000 7f1c30ff9640 10 monclient(hunting): dump: 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:17.970 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:17.971 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:17.971 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.972+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2213298666 0 0) 0x7f1c340abb60 con 0x7f1c340ae6e0 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.972+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 >> v1:172.21.15.32:6790/0 conn(0x7f1c341d5a30 legacy=0x7f1c341e0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 >> v1:172.21.15.32:6789/0 conn(0x7f1c340704e0 legacy=0x7f1c341d5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 1 monclient: found mon.b 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c340ab780 con 0x7f1c340ae6e0 2024-03-25T18:40:17.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 10 monclient: _finish_auth 0 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 5 monclient: authenticate success, global_id 4261 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 10 monclient: _renew_subs 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c34142800 con 0x7f1c340ae6e0 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 10 monclient: _renew_subs 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c34137380 con 0x7f1c340ae6e0 2024-03-25T18:40:17.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.973+0000 7f1c3a723640 1 librados: init done 2024-03-25T18:40:17.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.974+0000 7f1c3a723640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:17.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.974+0000 7f1c3a723640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:17.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.974+0000 7f1c3a723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:17.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:17.974+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c340ab9d0 con 0x7f1c340ae6e0 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.241+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c28001c40 con 0x7f1c340ae6e0 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.241+0000 7f1c30ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.242+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c28003a60 con 0x7f1c340ae6e0 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.242+0000 7f1c30ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.242+0000 7f1c30ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.242+0000 7f1c30ff9640 10 monclient: dump: 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:18.242 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.242+0000 7f1c327fc640 4 set_mon_vals no callback set 2024-03-25T18:40:18.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.243+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f1c28003c40 con 0x7f1c340ae6e0 2024-03-25T18:40:18.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.243+0000 7f1c33fff640 10 monclient: get_auth_request con 0x7f1c2006e0f0 auth_method 0 2024-03-25T18:40:18.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.243+0000 7f1c30ff9640 10 client.4261.objecter ms_handle_connect 0x7f1c2006e0f0 2024-03-25T18:40:18.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.248+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f1c28054940 con 0x7f1c340ae6e0 2024-03-25T18:40:18.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.248+0000 7f1c30ff9640 10 client.4261.objecter ms_dispatch 0x7f1c34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:40:18.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.248+0000 7f1c30ff9640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:40:18.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.248+0000 7f1c30ff9640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:40:18.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.248+0000 7f1c30ff9640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-25T18:40:18.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.249+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c28050450 con 0x7f1c340ae6e0 2024-03-25T18:40:18.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.249+0000 7f1c30ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:18.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.249+0000 7f1c30ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:40:18.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.475+0000 7f1c3a723640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:18.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.475+0000 7f1c3a723640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:18.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.475+0000 7f1c3a723640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:18.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.475+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c34074d80 con 0x7f1c340ae6e0 2024-03-25T18:40:18.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.498+0000 7f1c30ff9640 1 -- 172.21.15.32:0/748880835 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f1c34074d80 con 0x7f1c340ae6e0 2024-03-25T18:40:18.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.498+0000 7f1c30ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:18.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.498+0000 7f1c30ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:40:18.499 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:40:18.500 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 10 librados: watch_flush enter 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 10 librados: watch_flush exit 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 10 client.4261.objecter successfully canceled tick 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 >> v1:172.21.15.136:6800/1141502808 conn(0x7f1c2006e0f0 legacy=0x7f1c200705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 10 monclient: shutdown 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 >> v1:172.21.15.136:6789/0 conn(0x7f1c340ae6e0 legacy=0x7f1c34136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 shutdown_connections 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 >> 172.21.15.32:0/748880835 conn(0x7f1c3406b630 msgr2=0x7f1c3413c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 shutdown_connections 2024-03-25T18:40:18.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 1 -- 172.21.15.32:0/748880835 wait complete. 2024-03-25T18:40:18.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:18.500+0000 7f1c3a723640 1 librados: shutdown 2024-03-25T18:40:18.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:40:25.320 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.433+0000 7f22411ed640 10 monclient: get_monmap_and_config 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.433+0000 7f22411ed640 10 monclient: build_initial_monmap 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.433+0000 7f22411ed640 10 monclient: monmap: 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:40:25.435333+0000 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:40:25.435333+0000 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:25.435 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:40:25.436 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:40:25.436 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:25.437 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.434+0000 7f22411ed640 1 Processor -- start 2024-03-25T18:40:25.437 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.434+0000 7f22411ed640 1 -- start start 2024-03-25T18:40:25.437 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.434+0000 7f22411ed640 10 monclient: init 2024-03-25T18:40:25.437 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:25.437 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:40:25.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 10 monclient(hunting): picked mon.noname-b con 0x7f223c0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:25.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 10 monclient(hunting): picked mon.noname-a con 0x7f223c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:25.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 10 monclient(hunting): picked mon.noname-c con 0x7f223c1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:25.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c142520 con 0x7f223c13f710 2024-03-25T18:40:25.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c0b28a0 con 0x7f223c1430c0 2024-03-25T18:40:25.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c0b2a30 con 0x7f223c0a98f0 2024-03-25T18:40:25.439 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:25.439 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.435+0000 7f22411ed640 10 monclient(hunting): authenticate will time out at 6131.625239s 2024-03-25T18:40:25.444 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.443+0000 7f223b59e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f223c1430c0 0x7f223c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43056/0 (socket says 172.21.15.32:43056) 2024-03-25T18:40:25.444 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.443+0000 7f223b59e640 1 -- 172.21.15.32:0/2056917304 learned_addr learned my addr 172.21.15.32:0/2056917304 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:25.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.456+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4169488003 0 0) 0x7f223c0b28a0 con 0x7f223c1430c0 2024-03-25T18:40:25.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.456+0000 7f2239d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:40:25.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.456+0000 7f2239d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:25.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.456+0000 7f2239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:25.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.456+0000 7f2239d9b640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:40:25.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.456+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2224003520 con 0x7f223c1430c0 2024-03-25T18:40:25.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.459+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425656898 0 0) 0x7f223c0b2a30 con 0x7f223c0a98f0 2024-03-25T18:40:25.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.459+0000 7f2239d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:40:25.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.459+0000 7f2239d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:25.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.459+0000 7f2239d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:25.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.459+0000 7f2239d9b640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:40:25.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.459+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c0b28a0 con 0x7f223c0a98f0 2024-03-25T18:40:25.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2875399309 0 0) 0x7f2224003520 con 0x7f223c1430c0 2024-03-25T18:40:25.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 >> v1:172.21.15.32:6790/0 conn(0x7f223c0a98f0 legacy=0x7f223c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:25.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 >> v1:172.21.15.32:6789/0 conn(0x7f223c13f710 legacy=0x7f223c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:25.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 1 monclient: found mon.noname-c 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c0b44c0 con 0x7f223c1430c0 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 10 monclient: _finish_auth 0 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2234001a10 con 0x7f223c1430c0 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f22411ed640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f22411ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:40:25.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f2239d9b640 10 monclient: dump: 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:25.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.479+0000 7f22411ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2234001b70 con 0x7f223c1430c0 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f2239d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f2239d9b640 1 -- 172.21.15.32:0/2056917304 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2234001f80 con 0x7f223c1430c0 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f22411ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f22411ed640 4 set_mon_vals no callback set 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f2239d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f2239d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f2239d9b640 10 monclient: dump: 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:25.493 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:25.494 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:25.494 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:25.494 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:25.494 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:25.494 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f22411ed640 10 monclient: shutdown 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f22411ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.492+0000 7f22411ed640 1 -- 172.21.15.32:0/2056917304 >> v1:172.21.15.136:6789/0 conn(0x7f223c1430c0 legacy=0x7f223c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.493+0000 7f22411ed640 1 -- 172.21.15.32:0/2056917304 shutdown_connections 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.493+0000 7f22411ed640 1 -- 172.21.15.32:0/2056917304 >> 172.21.15.32:0/2056917304 conn(0x7f223c0b7ea0 msgr2=0x7f223c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.493+0000 7f22411ed640 1 -- 172.21.15.32:0/2056917304 shutdown_connections 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.493+0000 7f22411ed640 1 -- 172.21.15.32:0/2056917304 wait complete. 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.494+0000 7f22411ed640 10 monclient: build_initial_monmap 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.494+0000 7f22411ed640 10 monclient: monmap: 2024-03-25T18:40:25.495 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.494+0000 7f22411ed640 1 librados: starting msgr at 2024-03-25T18:40:25.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.494+0000 7f22411ed640 1 librados: starting objecter 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.494+0000 7f22411ed640 1 Processor -- start 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 1 -- start start 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 1 librados: setting wanted keys 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 1 librados: calling monclient init 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 10 monclient: init 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 10 monclient(hunting): picked mon.noname-c con 0x7f223c0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:25.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 10 monclient(hunting): picked mon.noname-a con 0x7f223c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:25.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 10 monclient(hunting): picked mon.noname-b con 0x7f223c1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:25.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c137c60 con 0x7f223c13f710 2024-03-25T18:40:25.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c139160 con 0x7f223c1430c0 2024-03-25T18:40:25.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f223c1380f0 con 0x7f223c0a98f0 2024-03-25T18:40:25.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:25.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.495+0000 7f22411ed640 10 monclient(hunting): authenticate will time out at 6131.685442s 2024-03-25T18:40:25.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.501+0000 7f223a59c640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f223c13f710 0x7f223c133070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53438/0 (socket says 172.21.15.32:53438) 2024-03-25T18:40:25.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.501+0000 7f223a59c640 1 -- 172.21.15.32:0/383595828 learned_addr learned my addr 172.21.15.32:0/383595828 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:25.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.502+0000 7f22237fe640 10 client.?.objecter ms_handle_connect 0x7f223c13f710 2024-03-25T18:40:25.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.502+0000 7f22237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:25.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.515+0000 7f22237fe640 10 client.?.objecter ms_handle_connect 0x7f223c0a98f0 2024-03-25T18:40:25.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.515+0000 7f22237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:25.519 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.518+0000 7f22237fe640 10 client.?.objecter ms_handle_connect 0x7f223c1430c0 2024-03-25T18:40:25.519 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.518+0000 7f22237fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:25.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.529+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3762701460 0 0) 0x7f223c139160 con 0x7f223c1430c0 2024-03-25T18:40:25.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.529+0000 7f22237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:25.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.529+0000 7f22237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:25.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.529+0000 7f22237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:25.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.529+0000 7f22237fe640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:40:25.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.529+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2228004eb0 con 0x7f223c1430c0 2024-03-25T18:40:25.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.531+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 644584964 0 0) 0x7f223c137c60 con 0x7f223c13f710 2024-03-25T18:40:25.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.531+0000 7f22237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:25.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.531+0000 7f22237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:25.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.531+0000 7f22237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:25.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.531+0000 7f22237fe640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:40:25.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.531+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c139160 con 0x7f223c13f710 2024-03-25T18:40:25.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.532+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2348221716 0 0) 0x7f223c1380f0 con 0x7f223c0a98f0 2024-03-25T18:40:25.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.532+0000 7f22237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:25.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.532+0000 7f22237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:25.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.532+0000 7f22237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:25.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.532+0000 7f22237fe640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:40:25.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.532+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f223c137c60 con 0x7f223c0a98f0 2024-03-25T18:40:25.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.543+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1494859428 0 0) 0x7f2228004eb0 con 0x7f223c1430c0 2024-03-25T18:40:25.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.543+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f223c1380f0 con 0x7f223c1430c0 2024-03-25T18:40:25.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.543+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2234001e80 con 0x7f223c1430c0 2024-03-25T18:40:25.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.543+0000 7f22237fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:25.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.543+0000 7f22237fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.543+0000 7f22237fe640 10 monclient(hunting): dump: 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:25.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.543+0000 7f22237fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2426407405 0 0) 0x7f223c1380f0 con 0x7f223c1430c0 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 >> v1:172.21.15.32:6790/0 conn(0x7f223c0a98f0 legacy=0x7f223c1dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 >> v1:172.21.15.32:6789/0 conn(0x7f223c13f710 legacy=0x7f223c133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 10 monclient: _finish_hunting 0 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 1 monclient: found mon.b 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c1397f0 con 0x7f223c1430c0 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 10 monclient: _finish_auth 0 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 10 monclient: _check_auth_tickets 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:25.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22411ed640 5 monclient: authenticate success, global_id 4267 2024-03-25T18:40:25.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22411ed640 10 monclient: _renew_subs 2024-03-25T18:40:25.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22411ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:25.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.549+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223c1db280 con 0x7f223c1430c0 2024-03-25T18:40:25.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 10 client.4267.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:40:25.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 10 monclient: _renew_subs 2024-03-25T18:40:25.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:25.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c133960 con 0x7f223c1430c0 2024-03-25T18:40:25.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 1 librados: init done 2024-03-25T18:40:25.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:25.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:25.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:25.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.550+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223c1db5d0 con 0x7f223c1430c0 2024-03-25T18:40:25.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.831+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2234002020 con 0x7f223c1430c0 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.831+0000 7f22237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.831+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2234004fb0 con 0x7f223c1430c0 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.831+0000 7f22237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.831+0000 7f22237fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.831+0000 7f22237fe640 10 monclient: dump: 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:25.961 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:25.962 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:25.962 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:25.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.831+0000 7f223959a640 4 set_mon_vals no callback set 2024-03-25T18:40:25.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.832+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f223401af80 con 0x7f223c1430c0 2024-03-25T18:40:25.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.832+0000 7f223ad9d640 10 monclient: get_auth_request con 0x7f222806ddf0 auth_method 0 2024-03-25T18:40:25.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.833+0000 7f22237fe640 10 client.4267.objecter ms_handle_connect 0x7f222806ddf0 2024-03-25T18:40:25.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.836+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f2234054af0 con 0x7f223c1430c0 2024-03-25T18:40:25.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.836+0000 7f22237fe640 10 client.4267.objecter ms_dispatch 0x7f223c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:40:25.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.836+0000 7f22237fe640 3 client.4267.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:40:25.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.836+0000 7f22237fe640 3 client.4267.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:40:25.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.836+0000 7f22237fe640 20 client.4267.objecter dump_active .. 0 homeless 2024-03-25T18:40:25.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.837+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2234050600 con 0x7f223c1430c0 2024-03-25T18:40:25.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.837+0000 7f22237fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:25.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:25.837+0000 7f22237fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:40:26.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.086+0000 7f22411ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:26.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.086+0000 7f22411ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:26.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.086+0000 7f22411ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:40:26.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.086+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f223c137e10 con 0x7f223c1430c0 2024-03-25T18:40:26.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.114+0000 7f22237fe640 1 -- 172.21.15.32:0/383595828 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f223c137e10 con 0x7f223c1430c0 2024-03-25T18:40:26.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.114+0000 7f22237fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:26.116 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.114+0000 7f22237fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:40:26.116 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:40:26.116 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:40:26.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 10 librados: watch_flush enter 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 10 librados: watch_flush exit 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 20 client.4267.objecter shutdown clearing up homeless session... 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 10 client.4267.objecter successfully canceled tick 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 >> v1:172.21.15.136:6800/1141502808 conn(0x7f222806ddf0 legacy=0x7f22280702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 10 monclient: shutdown 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 >> v1:172.21.15.136:6789/0 conn(0x7f223c1430c0 legacy=0x7f223c1e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 shutdown_connections 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 >> 172.21.15.32:0/383595828 conn(0x7f223c0b7ea0 msgr2=0x7f223c13d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:26.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 shutdown_connections 2024-03-25T18:40:26.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 1 -- 172.21.15.32:0/383595828 wait complete. 2024-03-25T18:40:26.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:26.117+0000 7f22411ed640 1 librados: shutdown 2024-03-25T18:40:26.129 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:40:27.159 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:39.487 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:27.168 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:39.496 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:27.212 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:39.540 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:27.223 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:39.551 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:27.227 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:39.555 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:27.235 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:39.552 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:27.252 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:39.569 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:27.280 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:39.608 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:27.295 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:39.612 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:32.935 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:40:33.051 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.050+0000 7f7726ee4640 10 monclient: get_monmap_and_config 2024-03-25T18:40:33.051 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.050+0000 7f7726ee4640 10 monclient: build_initial_monmap 2024-03-25T18:40:33.051 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.050+0000 7f7726ee4640 10 monclient: monmap: 2024-03-25T18:40:33.051 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:33.051 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:33.051 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:40:33.051816+0000 2024-03-25T18:40:33.052 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:40:33.051816+0000 2024-03-25T18:40:33.052 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:33.052 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:33.052 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:33.052 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:40:33.052 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:40:33.052 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:33.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 1 Processor -- start 2024-03-25T18:40:33.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 1 -- start start 2024-03-25T18:40:33.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 10 monclient: init 2024-03-25T18:40:33.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:33.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:40:33.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f77200a96e0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:33.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f77200b1b10 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:33.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f7720146060 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:33.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7720140040 con 0x7f77200b1b10 2024-03-25T18:40:33.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200b1fa0 con 0x7f77200a96e0 2024-03-25T18:40:33.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200b24c0 con 0x7f7720146060 2024-03-25T18:40:33.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:33.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.051+0000 7f7726ee4640 10 monclient(hunting): authenticate will time out at 6139.241381s 2024-03-25T18:40:33.060 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.060+0000 7f771ffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f77200b1b10 0x7f7720145950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44226/0 (socket says 172.21.15.32:44226) 2024-03-25T18:40:33.061 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.060+0000 7f771ffff640 1 -- 172.21.15.32:0/3740218713 learned_addr learned my addr 172.21.15.32:0/3740218713 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:33.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.071+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665937282 0 0) 0x7f7720140040 con 0x7f77200b1b10 2024-03-25T18:40:33.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.071+0000 7f771f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:33.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.071+0000 7f771f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:33.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.071+0000 7f771f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:33.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.071+0000 7f771f7fe640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:40:33.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.071+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7710003520 con 0x7f77200b1b10 2024-03-25T18:40:33.082 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.081+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957082202 0 0) 0x7f77200b24c0 con 0x7f7720146060 2024-03-25T18:40:33.082 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.081+0000 7f771f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:33.082 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.081+0000 7f771f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:33.082 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.081+0000 7f771f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:33.083 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.081+0000 7f771f7fe640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:40:33.083 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.081+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7720140040 con 0x7f7720146060 2024-03-25T18:40:33.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.085+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2962295439 0 0) 0x7f77200b1fa0 con 0x7f77200a96e0 2024-03-25T18:40:33.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.085+0000 7f771f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:33.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.085+0000 7f771f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:33.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.085+0000 7f771f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:33.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.085+0000 7f771f7fe640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:40:33.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.085+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77200b24c0 con 0x7f77200a96e0 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2295322186 0 0) 0x7f7710003520 con 0x7f77200b1b10 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 >> v1:172.21.15.32:6790/0 conn(0x7f7720146060 legacy=0x7f772014dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 >> v1:172.21.15.136:6789/0 conn(0x7f77200a96e0 legacy=0x7f77200b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77200b15b0 con 0x7f77200b1b10 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:33.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f770c001810 con 0x7f77200b1b10 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f771f7fe640 10 monclient: dump: 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f7726ee4640 5 monclient: authenticate success, global_id 4278 2024-03-25T18:40:33.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.090+0000 7f7726ee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:33.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.108+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f770c001970 con 0x7f77200b1b10 2024-03-25T18:40:33.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.108+0000 7f771f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:33.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.108+0000 7f771f7fe640 1 -- 172.21.15.32:0/3740218713 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f770c001d80 con 0x7f77200b1b10 2024-03-25T18:40:33.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.108+0000 7f7726ee4640 10 monclient: get_monmap_and_config success 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.108+0000 7f7726ee4640 4 set_mon_vals no callback set 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.108+0000 7f771f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.108+0000 7f771f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.108+0000 7f771f7fe640 10 monclient: dump: 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:33.110 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.109+0000 7f7726ee4640 10 monclient: shutdown 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.109+0000 7f7726ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.109+0000 7f7726ee4640 1 -- 172.21.15.32:0/3740218713 >> v1:172.21.15.32:6789/0 conn(0x7f77200b1b10 legacy=0x7f7720145950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.109+0000 7f7726ee4640 1 -- 172.21.15.32:0/3740218713 shutdown_connections 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.109+0000 7f7726ee4640 1 -- 172.21.15.32:0/3740218713 >> 172.21.15.32:0/3740218713 conn(0x7f77200b7ea0 msgr2=0x7f77200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.109+0000 7f7726ee4640 1 -- 172.21.15.32:0/3740218713 shutdown_connections 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.109+0000 7f7726ee4640 1 -- 172.21.15.32:0/3740218713 wait complete. 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 10 monclient: build_initial_monmap 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 10 monclient: monmap: 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:40:33.112 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 1 librados: starting msgr at 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 1 librados: starting objecter 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 1 Processor -- start 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 1 -- start start 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 1 librados: setting wanted keys 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 1 librados: calling monclient init 2024-03-25T18:40:33.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.110+0000 7f7726ee4640 10 monclient: init 2024-03-25T18:40:33.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:33.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:40:33.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 10 monclient(hunting): picked mon.noname-a con 0x7f77200a96e0 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:33.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 10 monclient(hunting): picked mon.noname-b con 0x7f7720146060 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:33.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 10 monclient(hunting): picked mon.noname-c con 0x7f77200ad250 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:33.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200acfb0 con 0x7f77200a96e0 2024-03-25T18:40:33.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f772013ae00 con 0x7f7720146060 2024-03-25T18:40:33.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77200aeaa0 con 0x7f77200ad250 2024-03-25T18:40:33.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:33.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.111+0000 7f7726ee4640 10 monclient(hunting): authenticate will time out at 6139.301351s 2024-03-25T18:40:33.116 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.115+0000 7f7725481640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f77200ad250 0x7f772013a6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:34758/0 (socket says 172.21.15.32:34758) 2024-03-25T18:40:33.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.115+0000 7f7725481640 1 -- 172.21.15.32:0/344036095 learned_addr learned my addr 172.21.15.32:0/344036095 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:33.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.121+0000 7f771d7fa640 10 client.?.objecter ms_handle_connect 0x7f77200ad250 2024-03-25T18:40:33.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.121+0000 7f771d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:33.126 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.125+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3738586587 0 0) 0x7f77200aeaa0 con 0x7f77200ad250 2024-03-25T18:40:33.126 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.125+0000 7f771d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:40:33.126 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.125+0000 7f771d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:33.126 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.125+0000 7f771d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:33.126 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.125+0000 7f771d7fa640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:40:33.126 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.125+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7704004eb0 con 0x7f77200ad250 2024-03-25T18:40:33.127 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.126+0000 7f771d7fa640 10 client.?.objecter ms_handle_connect 0x7f77200a96e0 2024-03-25T18:40:33.127 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.126+0000 7f771d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:33.130 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.129+0000 7f771d7fa640 10 client.?.objecter ms_handle_connect 0x7f7720146060 2024-03-25T18:40:33.130 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.129+0000 7f771d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:33.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.134+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 377144086 0 0) 0x7f7704004eb0 con 0x7f77200ad250 2024-03-25T18:40:33.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.134+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f77200aeaa0 con 0x7f77200ad250 2024-03-25T18:40:33.137 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.136+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7718001d90 con 0x7f77200ad250 2024-03-25T18:40:33.137 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.136+0000 7f771d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.136+0000 7f771d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.136+0000 7f771d7fa640 10 monclient(hunting): dump: 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:33.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.136+0000 7f771d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:40:33.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.140+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3666826412 0 0) 0x7f77200b24c0 con 0x7f77200a96e0 2024-03-25T18:40:33.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.140+0000 7f771d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:40:33.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.140+0000 7f771d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:33.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.140+0000 7f771d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:33.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.140+0000 7f771d7fa640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:40:33.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.140+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7704004eb0 con 0x7f77200a96e0 2024-03-25T18:40:33.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 917911066 0 0) 0x7f77200aeaa0 con 0x7f77200ad250 2024-03-25T18:40:33.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 >> v1:172.21.15.136:6789/0 conn(0x7f7720146060 legacy=0x7f7720139fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:33.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 >> v1:172.21.15.32:6789/0 conn(0x7f77200a96e0 legacy=0x7f77200af210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:33.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:40:33.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 1 monclient: found mon.c 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77200ac6b0 con 0x7f77200ad250 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f771d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 10 monclient: _renew_subs 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77200b0430 con 0x7f77200ad250 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 10 monclient: _renew_subs 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:33.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f77200afb00 con 0x7f77200ad250 2024-03-25T18:40:33.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.141+0000 7f7726ee4640 1 librados: init done 2024-03-25T18:40:33.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.142+0000 7f7726ee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:33.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.142+0000 7f7726ee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:33.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.142+0000 7f7726ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:33.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.142+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77200b07d0 con 0x7f77200ad250 2024-03-25T18:40:33.361 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.360+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7718003a10 con 0x7f77200ad250 2024-03-25T18:40:33.361 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.360+0000 7f771d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:33.361 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.360+0000 7f771effd640 4 set_mon_vals no callback set 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.447+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7718003e20 con 0x7f77200ad250 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.447+0000 7f771d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.447+0000 7f771d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.447+0000 7f771d7fa640 10 monclient: dump: 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:33.449 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:33.450 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:33.450 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:33.450 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:33.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.448+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f7718004000 con 0x7f77200ad250 2024-03-25T18:40:33.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.449+0000 7f7724c80640 10 monclient: get_auth_request con 0x7f770406dd10 auth_method 0 2024-03-25T18:40:33.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.449+0000 7f771d7fa640 10 client.4277.objecter ms_handle_connect 0x7f770406dd10 2024-03-25T18:40:33.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.456+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f7718054c50 con 0x7f77200ad250 2024-03-25T18:40:33.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.456+0000 7f771d7fa640 10 client.4277.objecter ms_dispatch 0x7f7720000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:40:33.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.456+0000 7f771d7fa640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:40:33.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.456+0000 7f771d7fa640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:40:33.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.456+0000 7f771d7fa640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T18:40:33.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.456+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7718050760 con 0x7f77200ad250 2024-03-25T18:40:33.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.457+0000 7f771d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:33.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.457+0000 7f771d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:40:33.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.682+0000 7f7726ee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:33.684 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.682+0000 7f7726ee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:33.684 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.682+0000 7f7726ee4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:33.684 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.682+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f77200afdb0 con 0x7f77200ad250 2024-03-25T18:40:33.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.707+0000 7f771d7fa640 1 -- 172.21.15.32:0/344036095 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f7718052690 con 0x7f77200ad250 2024-03-25T18:40:33.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.707+0000 7f771d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:33.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.707+0000 7f771d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:40:33.708 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:40:33.708 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 10 librados: watch_flush enter 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 10 librados: watch_flush exit 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 10 client.4277.objecter successfully canceled tick 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 >> v1:172.21.15.136:6800/1141502808 conn(0x7f770406dd10 legacy=0x7f77040701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 10 monclient: shutdown 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 >> v1:172.21.15.32:6790/0 conn(0x7f77200ad250 legacy=0x7f772013a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 shutdown_connections 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 >> 172.21.15.32:0/344036095 conn(0x7f77200b7ea0 msgr2=0x7f77200b8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.709+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 shutdown_connections 2024-03-25T18:40:33.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.710+0000 7f7726ee4640 1 -- 172.21.15.32:0/344036095 wait complete. 2024-03-25T18:40:33.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:33.710+0000 7f7726ee4640 1 librados: shutdown 2024-03-25T18:40:33.721 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:40:40.529 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:40:40.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.640+0000 7f610679d640 10 monclient: get_monmap_and_config 2024-03-25T18:40:40.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.640+0000 7f610679d640 10 monclient: build_initial_monmap 2024-03-25T18:40:40.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.640+0000 7f610679d640 10 monclient: monmap: 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:40:40.641973+0000 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:40:40.641973+0000 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:40:40.642 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:40.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.641+0000 7f610679d640 1 Processor -- start 2024-03-25T18:40:40.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.641+0000 7f610679d640 1 -- start start 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.641+0000 7f610679d640 10 monclient: init 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 10 monclient(hunting): picked mon.noname-a con 0x7f61000a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 10 monclient(hunting): picked mon.noname-c con 0x7f610013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 10 monclient(hunting): picked mon.noname-b con 0x7f61001430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6100142520 con 0x7f61000a98f0 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61000b28a0 con 0x7f610013f710 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61000b2a30 con 0x7f61001430c0 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:40.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.642+0000 7f610679d640 10 monclient(hunting): authenticate will time out at 6146.832147s 2024-03-25T18:40:40.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.653+0000 7f6104d3a640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f61001430c0 0x7f610014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:57502/0 (socket says 172.21.15.32:57502) 2024-03-25T18:40:40.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.653+0000 7f6104d3a640 1 -- 172.21.15.32:0/231368635 learned_addr learned my addr 172.21.15.32:0/231368635 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:40.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.661+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 962313813 0 0) 0x7f61000b2a30 con 0x7f61001430c0 2024-03-25T18:40:40.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.661+0000 7f60ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:40.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.661+0000 7f60ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:40.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.661+0000 7f60ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:40.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.661+0000 7f60ff7fe640 10 monclient(hunting): my global_id is 4280 2024-03-25T18:40:40.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.661+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60ec003520 con 0x7f61001430c0 2024-03-25T18:40:40.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.663+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1775993595 0 0) 0x7f61000b28a0 con 0x7f610013f710 2024-03-25T18:40:40.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.663+0000 7f60ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:40.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.663+0000 7f60ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:40.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.663+0000 7f60ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:40.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.663+0000 7f60ff7fe640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:40:40.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.663+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61000b2a30 con 0x7f610013f710 2024-03-25T18:40:40.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1039694329 0 0) 0x7f60ec003520 con 0x7f61001430c0 2024-03-25T18:40:40.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 >> v1:172.21.15.136:6789/0 conn(0x7f610013f710 legacy=0x7f61001429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:40.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 >> v1:172.21.15.32:6789/0 conn(0x7f61000a98f0 legacy=0x7f61000a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 1 monclient: found mon.noname-b 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000b44c0 con 0x7f61001430c0 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f60f8001a10 con 0x7f61001430c0 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f60ff7fe640 10 monclient: dump: 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:40.675 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:40.676 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:40.676 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:40.676 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:40.676 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:40.676 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:40.676 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:40.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f610679d640 5 monclient: authenticate success, global_id 4280 2024-03-25T18:40:40.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.673+0000 7f610679d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:40.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.705+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60f8001b70 con 0x7f61001430c0 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.705+0000 7f60ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.705+0000 7f60ff7fe640 1 -- 172.21.15.32:0/231368635 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f60f8001f80 con 0x7f61001430c0 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.705+0000 7f60ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.705+0000 7f60ff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.705+0000 7f60ff7fe640 10 monclient: dump: 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:40.707 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:40.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.705+0000 7f610679d640 10 monclient: get_monmap_and_config success 2024-03-25T18:40:40.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.705+0000 7f610679d640 4 set_mon_vals no callback set 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.706+0000 7f610679d640 10 monclient: shutdown 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.706+0000 7f610679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.706+0000 7f610679d640 1 -- 172.21.15.32:0/231368635 >> v1:172.21.15.32:6790/0 conn(0x7f61001430c0 legacy=0x7f610014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.706+0000 7f610679d640 1 -- 172.21.15.32:0/231368635 shutdown_connections 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.706+0000 7f610679d640 1 -- 172.21.15.32:0/231368635 >> 172.21.15.32:0/231368635 conn(0x7f61000b7ea0 msgr2=0x7f61000b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.706+0000 7f610679d640 1 -- 172.21.15.32:0/231368635 shutdown_connections 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.706+0000 7f610679d640 1 -- 172.21.15.32:0/231368635 wait complete. 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 10 monclient: build_initial_monmap 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 10 monclient: monmap: 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:40.710 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 1 librados: starting msgr at 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 1 librados: starting objecter 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 1 Processor -- start 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 1 -- start start 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 1 librados: setting wanted keys 2024-03-25T18:40:40.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 1 librados: calling monclient init 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.707+0000 7f610679d640 10 monclient: init 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 10 monclient(hunting): picked mon.noname-c con 0x7f61000a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 10 monclient(hunting): picked mon.noname-b con 0x7f610013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 10 monclient(hunting): picked mon.noname-a con 0x7f61001430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6100136ad0 con 0x7f61001430c0 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61000b7520 con 0x7f610013f710 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6100135810 con 0x7f61000a98f0 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:40.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.708+0000 7f610679d640 10 monclient(hunting): authenticate will time out at 6146.898107s 2024-03-25T18:40:40.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.718+0000 7f60f7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f610013f710 0x7f61000b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:37494/0 (socket says 172.21.15.32:37494) 2024-03-25T18:40:40.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.718+0000 7f60f7fff640 1 -- 172.21.15.32:0/894481772 learned_addr learned my addr 172.21.15.32:0/894481772 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:40.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.721+0000 7f60fd7fa640 10 client.?.objecter ms_handle_connect 0x7f610013f710 2024-03-25T18:40:40.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.721+0000 7f60fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:40.723 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.722+0000 7f60fd7fa640 10 client.?.objecter ms_handle_connect 0x7f61000a98f0 2024-03-25T18:40:40.723 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.722+0000 7f60fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:40.730 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.729+0000 7f60fd7fa640 10 client.?.objecter ms_handle_connect 0x7f61001430c0 2024-03-25T18:40:40.730 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.729+0000 7f60fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:40.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.736+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2820357503 0 0) 0x7f6100136ad0 con 0x7f61001430c0 2024-03-25T18:40:40.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.736+0000 7f60fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:40:40.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.736+0000 7f60fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:40.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.736+0000 7f60fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:40.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.736+0000 7f60fd7fa640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:40:40.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.736+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e0004eb0 con 0x7f61001430c0 2024-03-25T18:40:40.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.742+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268040898 0 0) 0x7f61000b7520 con 0x7f610013f710 2024-03-25T18:40:40.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.742+0000 7f60fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:40:40.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.742+0000 7f60fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:40.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.742+0000 7f60fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:40.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.742+0000 7f60fd7fa640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:40:40.743 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.742+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6100136ad0 con 0x7f610013f710 2024-03-25T18:40:40.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.750+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920988615 0 0) 0x7f6100135810 con 0x7f61000a98f0 2024-03-25T18:40:40.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.750+0000 7f60fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:40:40.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.750+0000 7f60fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:40.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.750+0000 7f60fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:40.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.750+0000 7f60fd7fa640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:40:40.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.750+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61000b7520 con 0x7f61000a98f0 2024-03-25T18:40:40.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.752+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2717276487 0 0) 0x7f60e0004eb0 con 0x7f61001430c0 2024-03-25T18:40:40.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.752+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6100135810 con 0x7f61001430c0 2024-03-25T18:40:40.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.752+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f60f8001ec0 con 0x7f61001430c0 2024-03-25T18:40:40.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.752+0000 7f60fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.752+0000 7f60fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.752+0000 7f60fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:40.754 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.752+0000 7f60fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:40:40.767 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.765+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1315543144 0 0) 0x7f6100135810 con 0x7f61001430c0 2024-03-25T18:40:40.767 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 >> v1:172.21.15.32:6790/0 conn(0x7f61000a98f0 legacy=0x7f6100135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:40.767 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 >> v1:172.21.15.136:6789/0 conn(0x7f610013f710 legacy=0x7f61000b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:40.767 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:40:40.767 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 1 monclient: found mon.a 2024-03-25T18:40:40.767 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:40.767 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61000b6180 con 0x7f61001430c0 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f60fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 5 monclient: authenticate success, global_id 4287 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 10 monclient: _renew_subs 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61000b7a40 con 0x7f61001430c0 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 10 monclient: _renew_subs 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6100131840 con 0x7f61001430c0 2024-03-25T18:40:40.768 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.766+0000 7f610679d640 1 librados: init done 2024-03-25T18:40:40.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.767+0000 7f610679d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:40.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.767+0000 7f610679d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:40.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.767+0000 7f610679d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:40.769 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:40.767+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6100136f60 con 0x7f61001430c0 2024-03-25T18:40:41.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.095+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60f80020e0 con 0x7f61001430c0 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.095+0000 7f60fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.095+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f60f8004e10 con 0x7f61001430c0 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.095+0000 7f60fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.095+0000 7f60fd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.095+0000 7f60fd7fa640 10 monclient: dump: 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:41.097 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:41.098 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:41.098 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:41.098 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.095+0000 7f60feffd640 4 set_mon_vals no callback set 2024-03-25T18:40:41.098 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.096+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f60f801ade0 con 0x7f61001430c0 2024-03-25T18:40:41.098 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.097+0000 7f60fffff640 10 monclient: get_auth_request con 0x7f60e006ddf0 auth_method 0 2024-03-25T18:40:41.098 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.097+0000 7f60fd7fa640 10 client.4287.objecter ms_handle_connect 0x7f60e006ddf0 2024-03-25T18:40:41.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.104+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f60f8054990 con 0x7f61001430c0 2024-03-25T18:40:41.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.104+0000 7f60fd7fa640 10 client.4287.objecter ms_dispatch 0x7f6100000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:40:41.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.104+0000 7f60fd7fa640 3 client.4287.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:40:41.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.104+0000 7f60fd7fa640 3 client.4287.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:40:41.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.104+0000 7f60fd7fa640 20 client.4287.objecter dump_active .. 0 homeless 2024-03-25T18:40:41.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.104+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60f80502d0 con 0x7f61001430c0 2024-03-25T18:40:41.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.104+0000 7f60fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:41.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.104+0000 7f60fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:40:41.362 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.361+0000 7f610679d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:41.362 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.361+0000 7f610679d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:41.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.361+0000 7f610679d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:41.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.361+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f61000b6310 con 0x7f61001430c0 2024-03-25T18:40:41.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.394+0000 7f60fd7fa640 1 -- 172.21.15.32:0/894481772 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f60f80523d0 con 0x7f61001430c0 2024-03-25T18:40:41.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.394+0000 7f60fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:41.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.394+0000 7f60fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:40:41.396 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:40:41.396 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 10 librados: watch_flush enter 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 10 librados: watch_flush exit 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 20 client.4287.objecter shutdown clearing up homeless session... 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 10 client.4287.objecter successfully canceled tick 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 >> v1:172.21.15.136:6800/1141502808 conn(0x7f60e006ddf0 legacy=0x7f60e00702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 10 monclient: shutdown 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 >> v1:172.21.15.32:6789/0 conn(0x7f61001430c0 legacy=0x7f6100130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 shutdown_connections 2024-03-25T18:40:41.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 >> 172.21.15.32:0/894481772 conn(0x7f61000b7ea0 msgr2=0x7f6100074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:41.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.396+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 shutdown_connections 2024-03-25T18:40:41.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.397+0000 7f610679d640 1 -- 172.21.15.32:0/894481772 wait complete. 2024-03-25T18:40:41.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:41.397+0000 7f610679d640 1 librados: shutdown 2024-03-25T18:40:41.410 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:40:47.167 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:59.494 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:47.221 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:59.549 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:47.265 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:01:59.592 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:47.266 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:59.594 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:47.279 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:59.607 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:47.298 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:59.614 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:47.305 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:59.622 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:47.310 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:01:59.638 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:47.331 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:01:59.648 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:40:48.216 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:40:48.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.327+0000 7f7722a9d640 10 monclient: get_monmap_and_config 2024-03-25T18:40:48.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.327+0000 7f7722a9d640 10 monclient: build_initial_monmap 2024-03-25T18:40:48.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.327+0000 7f7722a9d640 10 monclient: monmap: 2024-03-25T18:40:48.327 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:48.327 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:48.328 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:40:48.327879+0000 2024-03-25T18:40:48.328 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:40:48.327879+0000 2024-03-25T18:40:48.328 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:48.328 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:48.328 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:48.328 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:40:48.328 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:40:48.328 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:48.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.328+0000 7f7722a9d640 1 Processor -- start 2024-03-25T18:40:48.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.328+0000 7f7722a9d640 1 -- start start 2024-03-25T18:40:48.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.328+0000 7f7722a9d640 10 monclient: init 2024-03-25T18:40:48.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.328+0000 7f7722a9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:48.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.328+0000 7f7722a9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:40:48.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.328+0000 7f7722a9d640 10 monclient(hunting): picked mon.noname-b con 0x7f771c0b4f40 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:48.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.328+0000 7f7722a9d640 10 monclient(hunting): picked mon.noname-c con 0x7f771c13fd90 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:48.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.329+0000 7f7722a9d640 10 monclient(hunting): picked mon.noname-a con 0x7f771c0b4540 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:48.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.329+0000 7f7722a9d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c0aa0c0 con 0x7f771c0b4540 2024-03-25T18:40:48.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.329+0000 7f7722a9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c0ab170 con 0x7f771c13fd90 2024-03-25T18:40:48.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.329+0000 7f7722a9d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c0a9ba0 con 0x7f771c0b4f40 2024-03-25T18:40:48.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.329+0000 7f7722a9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:48.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.329+0000 7f7722a9d640 10 monclient(hunting): authenticate will time out at 6154.517581s 2024-03-25T18:40:48.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.338+0000 7f7720839640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f771c0b4f40 0x7f771c13f680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:55700/0 (socket says 172.21.15.32:55700) 2024-03-25T18:40:48.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.339+0000 7f7720839640 1 -- 172.21.15.32:0/2402904576 learned_addr learned my addr 172.21.15.32:0/2402904576 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:48.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.356+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1298589588 0 0) 0x7f771c0a9ba0 con 0x7f771c0b4f40 2024-03-25T18:40:48.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.356+0000 7f771b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:48.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.356+0000 7f771b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:48.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.356+0000 7f771b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:48.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.356+0000 7f771b7fe640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:40:48.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.356+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f770c003520 con 0x7f771c0b4f40 2024-03-25T18:40:48.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.375+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4254935710 0 0) 0x7f771c0aa0c0 con 0x7f771c0b4540 2024-03-25T18:40:48.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.376+0000 7f771b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:40:48.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.376+0000 7f771b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:48.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.376+0000 7f771b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:48.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.376+0000 7f771b7fe640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:40:48.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.376+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f771c0a9ba0 con 0x7f771c0b4540 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1125743185 0 0) 0x7f770c003520 con 0x7f771c0b4f40 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 >> v1:172.21.15.136:6789/0 conn(0x7f771c13fd90 legacy=0x7f771c1423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 >> v1:172.21.15.32:6789/0 conn(0x7f771c0b4540 legacy=0x7f771c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c0a9680 con 0x7f771c0b4f40 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:48.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7710001690 con 0x7f771c0b4f40 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f7722a9d640 5 monclient: authenticate success, global_id 4286 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f7722a9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f771b7fe640 10 monclient: dump: 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:48.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.377+0000 7f7722a9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:48.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.411+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f77100017f0 con 0x7f771c0b4f40 2024-03-25T18:40:48.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.411+0000 7f771b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.411+0000 7f771b7fe640 1 -- 172.21.15.32:0/2402904576 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7710001c00 con 0x7f771c0b4f40 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.411+0000 7f7722a9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.411+0000 7f7722a9d640 4 set_mon_vals no callback set 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.411+0000 7f771b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.411+0000 7f771b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.411+0000 7f771b7fe640 10 monclient: dump: 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:48.412 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:48.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.412+0000 7f7722a9d640 10 monclient: shutdown 2024-03-25T18:40:48.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.412+0000 7f7722a9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:48.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.412+0000 7f7722a9d640 1 -- 172.21.15.32:0/2402904576 >> v1:172.21.15.32:6790/0 conn(0x7f771c0b4f40 legacy=0x7f771c13f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:48.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.412+0000 7f7722a9d640 1 -- 172.21.15.32:0/2402904576 shutdown_connections 2024-03-25T18:40:48.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.412+0000 7f7722a9d640 1 -- 172.21.15.32:0/2402904576 >> 172.21.15.32:0/2402904576 conn(0x7f771c0617a0 msgr2=0x7f771c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:48.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.412+0000 7f7722a9d640 1 -- 172.21.15.32:0/2402904576 shutdown_connections 2024-03-25T18:40:48.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.412+0000 7f7722a9d640 1 -- 172.21.15.32:0/2402904576 wait complete. 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.413+0000 7f7722a9d640 10 monclient: build_initial_monmap 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.413+0000 7f7722a9d640 10 monclient: monmap: 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.413+0000 7f7722a9d640 1 librados: starting msgr at 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.413+0000 7f7722a9d640 1 librados: starting objecter 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.414+0000 7f7722a9d640 1 Processor -- start 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.414+0000 7f7722a9d640 1 -- start start 2024-03-25T18:40:48.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.414+0000 7f7722a9d640 1 librados: setting wanted keys 2024-03-25T18:40:48.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.414+0000 7f7722a9d640 1 librados: calling monclient init 2024-03-25T18:40:48.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.414+0000 7f7722a9d640 10 monclient: init 2024-03-25T18:40:48.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 10 monclient(hunting): picked mon.noname-a con 0x7f771c0b4540 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 10 monclient(hunting): picked mon.noname-c con 0x7f771c0b4f40 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 10 monclient(hunting): picked mon.noname-b con 0x7f771c13fd90 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c1d7890 con 0x7f771c0b4540 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c138b50 con 0x7f771c13fd90 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f771c138750 con 0x7f771c0b4f40 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:48.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.415+0000 7f7722a9d640 10 monclient(hunting): authenticate will time out at 6154.603874s 2024-03-25T18:40:48.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.426+0000 7f7720839640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f771c0b4540 0x7f771c138040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41998/0 (socket says 172.21.15.32:41998) 2024-03-25T18:40:48.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.426+0000 7f7720839640 1 -- 172.21.15.32:0/1710141065 learned_addr learned my addr 172.21.15.32:0/1710141065 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:48.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.427+0000 7f77197fa640 10 client.?.objecter ms_handle_connect 0x7f771c13fd90 2024-03-25T18:40:48.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.427+0000 7f77197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:48.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.428+0000 7f77197fa640 10 client.?.objecter ms_handle_connect 0x7f771c0b4540 2024-03-25T18:40:48.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.428+0000 7f77197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:48.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.432+0000 7f77197fa640 10 client.?.objecter ms_handle_connect 0x7f771c0b4f40 2024-03-25T18:40:48.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.432+0000 7f77197fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:48.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.435+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4194117072 0 0) 0x7f771c138b50 con 0x7f771c13fd90 2024-03-25T18:40:48.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.435+0000 7f77197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:40:48.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.435+0000 7f77197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:48.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.435+0000 7f77197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:48.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.435+0000 7f77197fa640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:40:48.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.435+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7700004eb0 con 0x7f771c13fd90 2024-03-25T18:40:48.445 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.444+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3076577142 0 0) 0x7f771c1d7890 con 0x7f771c0b4540 2024-03-25T18:40:48.445 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.444+0000 7f77197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:40:48.445 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.444+0000 7f77197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:48.445 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.444+0000 7f77197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:48.445 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.444+0000 7f77197fa640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:40:48.445 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.444+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f771c138b50 con 0x7f771c0b4540 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.450+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1635220746 0 0) 0x7f7700004eb0 con 0x7f771c13fd90 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.450+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f771c1d7890 con 0x7f771c13fd90 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.450+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7714001cb0 con 0x7f771c13fd90 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.450+0000 7f77197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.450+0000 7f77197fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.450+0000 7f77197fa640 10 monclient(hunting): dump: 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:48.451 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:48.452 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:48.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.450+0000 7f77197fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:40:48.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.452+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1527392829 0 0) 0x7f771c138750 con 0x7f771c0b4f40 2024-03-25T18:40:48.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.452+0000 7f77197fa640 10 monclient(hunting): _negotiate 2024-03-25T18:40:48.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.452+0000 7f77197fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:48.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.452+0000 7f77197fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:48.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.452+0000 7f77197fa640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:40:48.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.452+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7700004eb0 con 0x7f771c0b4f40 2024-03-25T18:40:48.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.456+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2665939857 0 0) 0x7f771c138b50 con 0x7f771c0b4540 2024-03-25T18:40:48.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.456+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f771c138750 con 0x7f771c0b4540 2024-03-25T18:40:48.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.456+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7710001bc0 con 0x7f771c0b4540 2024-03-25T18:40:48.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.456+0000 7f77197fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.456+0000 7f77197fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.456+0000 7f77197fa640 10 monclient(hunting): dump: 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:48.457 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:48.464 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3132677192 0 0) 0x7f771c138750 con 0x7f771c0b4540 2024-03-25T18:40:48.464 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 >> v1:172.21.15.32:6790/0 conn(0x7f771c0b4f40 legacy=0x7f771c1d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 >> v1:172.21.15.136:6789/0 conn(0x7f771c13fd90 legacy=0x7f771c137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 10 monclient: _finish_hunting 0 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 1 monclient: found mon.a 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f771c1388e0 con 0x7f771c0b4540 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 10 monclient: _finish_auth 0 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 10 monclient: _check_auth_tickets 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f77197fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 5 monclient: authenticate success, global_id 4293 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 10 monclient: _renew_subs 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f771c0ae4a0 con 0x7f771c0b4540 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:40:48.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 10 monclient: _renew_subs 2024-03-25T18:40:48.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:48.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f771c1d8610 con 0x7f771c0b4540 2024-03-25T18:40:48.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.464+0000 7f7722a9d640 1 librados: init done 2024-03-25T18:40:48.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.465+0000 7f7722a9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:48.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.465+0000 7f7722a9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:48.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.465+0000 7f7722a9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:48.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.465+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f771c0627b0 con 0x7f771c0b4540 2024-03-25T18:40:48.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.774+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7710004a90 con 0x7f771c0b4540 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.774+0000 7f77197fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.774+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7710004da0 con 0x7f771c0b4540 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.774+0000 7f77197fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.774+0000 7f77197fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.774+0000 7f77197fa640 10 monclient: dump: 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:48.775 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:48.776 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:48.776 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:48.776 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:48.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.774+0000 7f771affd640 4 set_mon_vals no callback set 2024-03-25T18:40:48.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.775+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f771001ad70 con 0x7f771c0b4540 2024-03-25T18:40:48.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.776+0000 7f771bfff640 10 monclient: get_auth_request con 0x7f770006df70 auth_method 0 2024-03-25T18:40:48.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.776+0000 7f77197fa640 10 client.4293.objecter ms_handle_connect 0x7f770006df70 2024-03-25T18:40:48.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.781+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f7710054930 con 0x7f771c0b4540 2024-03-25T18:40:48.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.781+0000 7f77197fa640 10 client.4293.objecter ms_dispatch 0x7f771c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:40:48.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.781+0000 7f77197fa640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:40:48.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.781+0000 7f77197fa640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:40:48.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.781+0000 7f77197fa640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T18:40:48.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.781+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7710050270 con 0x7f771c0b4540 2024-03-25T18:40:48.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.781+0000 7f77197fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:48.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:48.781+0000 7f77197fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:40:49.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.044+0000 7f7722a9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:49.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.044+0000 7f7722a9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:49.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.044+0000 7f7722a9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:40:49.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.044+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f771c13a440 con 0x7f771c0b4540 2024-03-25T18:40:49.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.075+0000 7f77197fa640 1 -- 172.21.15.32:0/1710141065 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f7710052370 con 0x7f771c0b4540 2024-03-25T18:40:49.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.075+0000 7f77197fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:49.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.075+0000 7f77197fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:40:49.076 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:40:49.076 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 10 librados: watch_flush enter 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 10 librados: watch_flush exit 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 10 client.4293.objecter successfully canceled tick 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 >> v1:172.21.15.136:6800/1141502808 conn(0x7f770006df70 legacy=0x7f7700070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 10 monclient: shutdown 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 >> v1:172.21.15.32:6789/0 conn(0x7f771c0b4540 legacy=0x7f771c138040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 shutdown_connections 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 >> 172.21.15.32:0/1710141065 conn(0x7f771c0617a0 msgr2=0x7f771c0b75f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 shutdown_connections 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 1 -- 172.21.15.32:0/1710141065 wait complete. 2024-03-25T18:40:49.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:49.078+0000 7f7722a9d640 1 librados: shutdown 2024-03-25T18:40:49.090 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:40:55.896 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.011+0000 7fb987b3d640 10 monclient: get_monmap_and_config 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.011+0000 7fb987b3d640 10 monclient: build_initial_monmap 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.011+0000 7fb987b3d640 10 monclient: monmap: 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:40:56.011568+0000 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:40:56.011568+0000 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:56.011 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:56.012 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:56.012 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:40:56.012 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:40:56.012 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:56.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.011+0000 7fb987b3d640 1 Processor -- start 2024-03-25T18:40:56.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 1 -- start start 2024-03-25T18:40:56.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 10 monclient: init 2024-03-25T18:40:56.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:56.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:40:56.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 10 monclient(hunting): picked mon.noname-a con 0x7fb9800a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:56.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 10 monclient(hunting): picked mon.noname-c con 0x7fb98013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:56.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9801430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:56.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb980142520 con 0x7fb9800a98f0 2024-03-25T18:40:56.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800b28a0 con 0x7fb98013f710 2024-03-25T18:40:56.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800b2a30 con 0x7fb9801430c0 2024-03-25T18:40:56.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:56.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.012+0000 7fb987b3d640 10 monclient(hunting): authenticate will time out at 6162.201233s 2024-03-25T18:40:56.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.022+0000 7fb9858d9640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb9800a98f0 0x7fb9800a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46580/0 (socket says 172.21.15.32:46580) 2024-03-25T18:40:56.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.022+0000 7fb9858d9640 1 -- 172.21.15.32:0/42350943 learned_addr learned my addr 172.21.15.32:0/42350943 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:56.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.035+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3089940711 0 0) 0x7fb980142520 con 0x7fb9800a98f0 2024-03-25T18:40:56.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.035+0000 7fb9848d7640 10 monclient(hunting): _negotiate 2024-03-25T18:40:56.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.035+0000 7fb9848d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:56.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.035+0000 7fb9848d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:56.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.035+0000 7fb9848d7640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:40:56.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.035+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb96c003520 con 0x7fb9800a98f0 2024-03-25T18:40:56.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.038+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737980513 0 0) 0x7fb9800b28a0 con 0x7fb98013f710 2024-03-25T18:40:56.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.038+0000 7fb9848d7640 10 monclient(hunting): _negotiate 2024-03-25T18:40:56.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.038+0000 7fb9848d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:56.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.038+0000 7fb9848d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:56.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.038+0000 7fb9848d7640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:40:56.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.038+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb980142520 con 0x7fb98013f710 2024-03-25T18:40:56.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.043+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380064500 0 0) 0x7fb9800b2a30 con 0x7fb9801430c0 2024-03-25T18:40:56.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.043+0000 7fb9848d7640 10 monclient(hunting): _negotiate 2024-03-25T18:40:56.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.043+0000 7fb9848d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:56.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.043+0000 7fb9848d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:56.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.043+0000 7fb9848d7640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:40:56.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.043+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9800b28a0 con 0x7fb9801430c0 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2784851137 0 0) 0x7fb9800b28a0 con 0x7fb9801430c0 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 >> v1:172.21.15.136:6789/0 conn(0x7fb98013f710 legacy=0x7fb9801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 >> v1:172.21.15.32:6789/0 conn(0x7fb9800a98f0 legacy=0x7fb9800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 10 monclient: _finish_hunting 0 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 1 monclient: found mon.noname-b 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800b44c0 con 0x7fb9801430c0 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 10 monclient: _finish_auth 0 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 10 monclient: _check_auth_tickets 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb978001930 con 0x7fb9801430c0 2024-03-25T18:40:56.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb987b3d640 5 monclient: authenticate success, global_id 4292 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb987b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb9848d7640 10 monclient: dump: 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:56.054 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:56.057 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:56.057 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:56.057 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:56.057 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:56.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.052+0000 7fb987b3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb978001a90 con 0x7fb9801430c0 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb9848d7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb9848d7640 1 -- 172.21.15.32:0/42350943 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb978001ee0 con 0x7fb9801430c0 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb9848d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb9848d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb9848d7640 10 monclient: dump: 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:56.062 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb987b3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:40:56.063 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb987b3d640 4 set_mon_vals no callback set 2024-03-25T18:40:56.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb987b3d640 10 monclient: shutdown 2024-03-25T18:40:56.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb987b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:56.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.062+0000 7fb987b3d640 1 -- 172.21.15.32:0/42350943 >> v1:172.21.15.32:6790/0 conn(0x7fb9801430c0 legacy=0x7fb98014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:56.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.063+0000 7fb987b3d640 1 -- 172.21.15.32:0/42350943 shutdown_connections 2024-03-25T18:40:56.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.063+0000 7fb987b3d640 1 -- 172.21.15.32:0/42350943 >> 172.21.15.32:0/42350943 conn(0x7fb9800b7ea0 msgr2=0x7fb9800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.063+0000 7fb987b3d640 1 -- 172.21.15.32:0/42350943 shutdown_connections 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.063+0000 7fb987b3d640 1 -- 172.21.15.32:0/42350943 wait complete. 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.063+0000 7fb987b3d640 10 monclient: build_initial_monmap 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.063+0000 7fb987b3d640 10 monclient: monmap: 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:40:56.066 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 librados: starting msgr at 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 librados: starting objecter 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 Processor -- start 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 -- start start 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 librados: setting wanted keys 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 librados: calling monclient init 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 10 monclient: init 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 10 monclient(hunting): picked mon.noname-b con 0x7fb9800a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 10 monclient(hunting): picked mon.noname-a con 0x7fb98013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:40:56.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 10 monclient(hunting): picked mon.noname-c con 0x7fb9801430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:40:56.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9801395e0 con 0x7fb98013f710 2024-03-25T18:40:56.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9801398c0 con 0x7fb9800a98f0 2024-03-25T18:40:56.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9800b0cc0 con 0x7fb9801430c0 2024-03-25T18:40:56.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.064+0000 7fb987b3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:40:56.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.065+0000 7fb987b3d640 10 monclient(hunting): authenticate will time out at 6162.253522s 2024-03-25T18:40:56.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.075+0000 7fb9860da640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fb9801430c0 0x7fb9801d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:60392/0 (socket says 172.21.15.32:60392) 2024-03-25T18:40:56.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.076+0000 7fb9860da640 1 -- 172.21.15.32:0/804716390 learned_addr learned my addr 172.21.15.32:0/804716390 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:40:56.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.077+0000 7fb9767fc640 10 client.?.objecter ms_handle_connect 0x7fb9801430c0 2024-03-25T18:40:56.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.077+0000 7fb9767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:56.089 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.089+0000 7fb9767fc640 10 client.?.objecter ms_handle_connect 0x7fb9800a98f0 2024-03-25T18:40:56.089 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.089+0000 7fb9767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:56.094 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.094+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2417133309 0 0) 0x7fb9800b0cc0 con 0x7fb9801430c0 2024-03-25T18:40:56.094 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.094+0000 7fb9767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:40:56.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.094+0000 7fb9767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:56.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.094+0000 7fb9767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:56.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.094+0000 7fb9767fc640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:40:56.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.094+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb964004eb0 con 0x7fb9801430c0 2024-03-25T18:40:56.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.096+0000 7fb9767fc640 10 client.?.objecter ms_handle_connect 0x7fb98013f710 2024-03-25T18:40:56.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.096+0000 7fb9767fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:40:56.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.106+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2052531287 0 0) 0x7fb9801395e0 con 0x7fb98013f710 2024-03-25T18:40:56.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.107+0000 7fb9767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:40:56.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.107+0000 7fb9767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:56.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.107+0000 7fb9767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:56.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.107+0000 7fb9767fc640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:40:56.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.107+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9800b0cc0 con 0x7fb98013f710 2024-03-25T18:40:56.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.108+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1135564934 0 0) 0x7fb9801398c0 con 0x7fb9800a98f0 2024-03-25T18:40:56.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.108+0000 7fb9767fc640 10 monclient(hunting): _negotiate 2024-03-25T18:40:56.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.108+0000 7fb9767fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:40:56.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.108+0000 7fb9767fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:40:56.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.108+0000 7fb9767fc640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:40:56.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.108+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9801395e0 con 0x7fb9800a98f0 2024-03-25T18:40:56.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.123+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3705895397 0 0) 0x7fb9801395e0 con 0x7fb9800a98f0 2024-03-25T18:40:56.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.123+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9801398c0 con 0x7fb9800a98f0 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.123+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb97c001950 con 0x7fb9800a98f0 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.123+0000 7fb9767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.123+0000 7fb9767fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.123+0000 7fb9767fc640 10 monclient(hunting): dump: 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:56.124 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:56.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.123+0000 7fb9767fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:40:56.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.125+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481634029 0 0) 0x7fb964004eb0 con 0x7fb9801430c0 2024-03-25T18:40:56.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.125+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9801395e0 con 0x7fb9801430c0 2024-03-25T18:40:56.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.125+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb978001ea0 con 0x7fb9801430c0 2024-03-25T18:40:56.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.125+0000 7fb9767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:56.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.125+0000 7fb9767fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:40:56.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.125+0000 7fb9767fc640 10 monclient(hunting): dump: 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:56.126 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:56.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2147103120 0 0) 0x7fb9801395e0 con 0x7fb9801430c0 2024-03-25T18:40:56.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 >> v1:172.21.15.136:6789/0 conn(0x7fb9800a98f0 legacy=0x7fb9801d4450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 >> v1:172.21.15.32:6789/0 conn(0x7fb98013f710 legacy=0x7fb9800affb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 10 monclient: _finish_hunting 0 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 1 monclient: found mon.c 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9800b0e50 con 0x7fb9801430c0 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 10 monclient: _finish_auth 0 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 10 monclient: _check_auth_tickets 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb9767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 5 monclient: authenticate success, global_id 4295 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 10 monclient: _renew_subs 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:56.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9800b06c0 con 0x7fb9801430c0 2024-03-25T18:40:56.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:40:56.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 10 monclient: _renew_subs 2024-03-25T18:40:56.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:56.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9801d7b00 con 0x7fb9801430c0 2024-03-25T18:40:56.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.145+0000 7fb987b3d640 1 librados: init done 2024-03-25T18:40:56.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.146+0000 7fb987b3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:56.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.146+0000 7fb987b3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:56.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.146+0000 7fb987b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:56.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.146+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9800b1060 con 0x7fb9801430c0 2024-03-25T18:40:56.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.468+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb978003c20 con 0x7fb9801430c0 2024-03-25T18:40:56.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.468+0000 7fb9767fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:40:56.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.468+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb978002100 con 0x7fb9801430c0 2024-03-25T18:40:56.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.468+0000 7fb9767fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:40:56.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.468+0000 7fb9767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:40:56.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.468+0000 7fb9767fc640 10 monclient: dump: 2024-03-25T18:40:56.469 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:40:56.469 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:40:56.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.468+0000 7fb977fff640 4 set_mon_vals no callback set 2024-03-25T18:40:56.471 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.469+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fb978004070 con 0x7fb9801430c0 2024-03-25T18:40:56.471 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.470+0000 7fb9858d9640 10 monclient: get_auth_request con 0x7fb96406dfb0 auth_method 0 2024-03-25T18:40:56.471 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.470+0000 7fb9767fc640 10 client.4295.objecter ms_handle_connect 0x7fb96406dfb0 2024-03-25T18:40:56.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.479+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fb978054cd0 con 0x7fb9801430c0 2024-03-25T18:40:56.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.479+0000 7fb9767fc640 10 client.4295.objecter ms_dispatch 0x7fb980000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:40:56.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.479+0000 7fb9767fc640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:40:56.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.479+0000 7fb9767fc640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:40:56.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.479+0000 7fb9767fc640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-25T18:40:56.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.480+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb978050610 con 0x7fb9801430c0 2024-03-25T18:40:56.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.480+0000 7fb9767fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:40:56.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.480+0000 7fb9767fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:40:56.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.709+0000 7fb987b3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:56.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.709+0000 7fb987b3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:56.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.709+0000 7fb987b3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:40:56.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.709+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9800b3ef0 con 0x7fb9801430c0 2024-03-25T18:40:56.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.742+0000 7fb9767fc640 1 -- 172.21.15.32:0/804716390 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fb978052710 con 0x7fb9801430c0 2024-03-25T18:40:56.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.742+0000 7fb9767fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:40:56.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.742+0000 7fb9767fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:40:56.743 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:40:56.743 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:40:56.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.744+0000 7fb987b3d640 10 librados: watch_flush enter 2024-03-25T18:40:56.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.744+0000 7fb987b3d640 10 librados: watch_flush exit 2024-03-25T18:40:56.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-25T18:40:56.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 10 client.4295.objecter successfully canceled tick 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 >> v1:172.21.15.136:6800/1141502808 conn(0x7fb96406dfb0 legacy=0x7fb964070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 10 monclient: shutdown 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 >> v1:172.21.15.32:6790/0 conn(0x7fb9801430c0 legacy=0x7fb9801d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 shutdown_connections 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 >> 172.21.15.32:0/804716390 conn(0x7fb9800b7ea0 msgr2=0x7fb980074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 shutdown_connections 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 1 -- 172.21.15.32:0/804716390 wait complete. 2024-03-25T18:40:56.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:40:56.745+0000 7fb987b3d640 1 librados: shutdown 2024-03-25T18:40:56.757 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:41:03.565 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:41:03.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.678+0000 7f9abba86640 10 monclient: get_monmap_and_config 2024-03-25T18:41:03.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.678+0000 7f9abba86640 10 monclient: build_initial_monmap 2024-03-25T18:41:03.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.678+0000 7f9abba86640 10 monclient: monmap: 2024-03-25T18:41:03.679 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:03.679 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:03.679 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:41:03.679334+0000 2024-03-25T18:41:03.679 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:41:03.679334+0000 2024-03-25T18:41:03.680 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:03.680 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:03.680 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:03.680 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:41:03.680 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:41:03.680 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:03.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.679+0000 7f9abba86640 1 Processor -- start 2024-03-25T18:41:03.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.679+0000 7f9abba86640 1 -- start start 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.679+0000 7f9abba86640 10 monclient: init 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 10 monclient(hunting): picked mon.noname-a con 0x7f9ab40a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 10 monclient(hunting): picked mon.noname-c con 0x7f9ab413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab41430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab4142520 con 0x7f9ab40a98f0 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab40b28a0 con 0x7f9ab413f710 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab40b2a30 con 0x7f9ab41430c0 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:03.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.680+0000 7f9abba86640 10 monclient(hunting): authenticate will time out at 6169.869493s 2024-03-25T18:41:03.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.693+0000 7f9ab9021640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9ab413f710 0x7f9ab41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:48802/0 (socket says 172.21.15.32:48802) 2024-03-25T18:41:03.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.693+0000 7f9ab9021640 1 -- 172.21.15.32:0/2884224033 learned_addr learned my addr 172.21.15.32:0/2884224033 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:03.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.707+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628623078 0 0) 0x7f9ab40b28a0 con 0x7f9ab413f710 2024-03-25T18:41:03.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.707+0000 7f9ab8820640 10 monclient(hunting): _negotiate 2024-03-25T18:41:03.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.707+0000 7f9ab8820640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:03.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.707+0000 7f9ab8820640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:03.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.707+0000 7f9ab8820640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:41:03.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.707+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9aa0003520 con 0x7f9ab413f710 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3473736377 0 0) 0x7f9aa0003520 con 0x7f9ab413f710 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 >> v1:172.21.15.32:6790/0 conn(0x7f9ab41430c0 legacy=0x7f9ab414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 >> v1:172.21.15.32:6789/0 conn(0x7f9ab40a98f0 legacy=0x7f9ab40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 10 monclient: _finish_hunting 0 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 1 monclient: found mon.noname-c 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab40b44c0 con 0x7f9ab413f710 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 10 monclient: _finish_auth 0 2024-03-25T18:41:03.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 10 monclient: _check_auth_tickets 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9aa4001690 con 0x7f9ab413f710 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9abba86640 5 monclient: authenticate success, global_id 4291 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9abba86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9ab8820640 10 monclient: dump: 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.716 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:03.717 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:03.717 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:03.717 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:03.717 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:03.717 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:03.717 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.714+0000 7f9abba86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa4001860 con 0x7f9ab413f710 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9ab8820640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9ab8820640 1 -- 172.21.15.32:0/2884224033 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9aa4001c70 con 0x7f9ab413f710 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9ab8820640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9ab8820640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9ab8820640 10 monclient: dump: 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:03.731 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9abba86640 10 monclient: get_monmap_and_config success 2024-03-25T18:41:03.732 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9abba86640 4 set_mon_vals no callback set 2024-03-25T18:41:03.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9abba86640 10 monclient: shutdown 2024-03-25T18:41:03.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9abba86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:03.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.731+0000 7f9abba86640 1 -- 172.21.15.32:0/2884224033 >> v1:172.21.15.136:6789/0 conn(0x7f9ab413f710 legacy=0x7f9ab41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:03.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.732+0000 7f9abba86640 1 -- 172.21.15.32:0/2884224033 shutdown_connections 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.732+0000 7f9abba86640 1 -- 172.21.15.32:0/2884224033 >> 172.21.15.32:0/2884224033 conn(0x7f9ab40b7ea0 msgr2=0x7f9ab40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.732+0000 7f9abba86640 1 -- 172.21.15.32:0/2884224033 shutdown_connections 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.732+0000 7f9abba86640 1 -- 172.21.15.32:0/2884224033 wait complete. 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.732+0000 7f9abba86640 10 monclient: build_initial_monmap 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.732+0000 7f9abba86640 10 monclient: monmap: 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:41:03.735 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 librados: starting msgr at 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 librados: starting objecter 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 Processor -- start 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 -- start start 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 librados: setting wanted keys 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 librados: calling monclient init 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 10 monclient: init 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:41:03.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 10 monclient(hunting): picked mon.noname-c con 0x7f9ab40a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:03.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 10 monclient(hunting): picked mon.noname-b con 0x7f9ab413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:03.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 10 monclient(hunting): picked mon.noname-a con 0x7f9ab41430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:03.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab41d77a0 con 0x7f9ab41430c0 2024-03-25T18:41:03.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab41d8340 con 0x7f9ab413f710 2024-03-25T18:41:03.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ab40abb90 con 0x7f9ab40a98f0 2024-03-25T18:41:03.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:03.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.733+0000 7f9abba86640 10 monclient(hunting): authenticate will time out at 6169.922479s 2024-03-25T18:41:03.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.742+0000 7f9ab9822640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f9ab40a98f0 0x7f9ab4138610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:36226/0 (socket says 172.21.15.32:36226) 2024-03-25T18:41:03.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.742+0000 7f9ab9822640 1 -- 172.21.15.32:0/3158534371 learned_addr learned my addr 172.21.15.32:0/3158534371 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:03.744 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.744+0000 7f9aaa7fc640 10 client.?.objecter ms_handle_connect 0x7f9ab41430c0 2024-03-25T18:41:03.744 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.744+0000 7f9aaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:03.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.745+0000 7f9aaa7fc640 10 client.?.objecter ms_handle_connect 0x7f9ab40a98f0 2024-03-25T18:41:03.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.745+0000 7f9aaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:03.747 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.747+0000 7f9aaa7fc640 10 client.?.objecter ms_handle_connect 0x7f9ab413f710 2024-03-25T18:41:03.747 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.747+0000 7f9aaa7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:03.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.753+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818532512 0 0) 0x7f9ab41d8340 con 0x7f9ab413f710 2024-03-25T18:41:03.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.753+0000 7f9aaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:03.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.753+0000 7f9aaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:03.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.753+0000 7f9aaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:03.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.753+0000 7f9aaa7fc640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:41:03.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.753+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9a98004eb0 con 0x7f9ab413f710 2024-03-25T18:41:03.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.756+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1554900534 0 0) 0x7f9ab40abb90 con 0x7f9ab40a98f0 2024-03-25T18:41:03.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.756+0000 7f9aaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:03.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.756+0000 7f9aaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:03.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.756+0000 7f9aaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:03.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.756+0000 7f9aaa7fc640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:41:03.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.756+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab41d8340 con 0x7f9ab40a98f0 2024-03-25T18:41:03.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.763+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855450176 0 0) 0x7f9ab41d77a0 con 0x7f9ab41430c0 2024-03-25T18:41:03.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.763+0000 7f9aaa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:03.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.763+0000 7f9aaa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:03.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.763+0000 7f9aaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:03.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.763+0000 7f9aaa7fc640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:41:03.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.763+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ab40abb90 con 0x7f9ab41430c0 2024-03-25T18:41:03.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.776+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3778199525 0 0) 0x7f9a98004eb0 con 0x7f9ab413f710 2024-03-25T18:41:03.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.776+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab41d77a0 con 0x7f9ab413f710 2024-03-25T18:41:03.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.776+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9aa4001b70 con 0x7f9ab413f710 2024-03-25T18:41:03.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.776+0000 7f9aaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:03.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.776+0000 7f9aaa7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:41:03.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.776+0000 7f9aaa7fc640 10 monclient(hunting): dump: 2024-03-25T18:41:03.776 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:03.776 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:03.777 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.776+0000 7f9aaa7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:41:03.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.778+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 534343824 0 0) 0x7f9ab41d8340 con 0x7f9ab40a98f0 2024-03-25T18:41:03.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.778+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9a98004eb0 con 0x7f9ab40a98f0 2024-03-25T18:41:03.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.778+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9ab00019b0 con 0x7f9ab40a98f0 2024-03-25T18:41:03.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.778+0000 7f9aaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:03.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.778+0000 7f9aaa7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.778+0000 7f9aaa7fc640 10 monclient(hunting): dump: 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:03.779 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.783+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2113443336 0 0) 0x7f9ab40abb90 con 0x7f9ab41430c0 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.783+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ab41d8340 con 0x7f9ab41430c0 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.783+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9aac001a10 con 0x7f9ab41430c0 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.783+0000 7f9aaa7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.783+0000 7f9aaa7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.783+0000 7f9aaa7fc640 10 monclient(hunting): dump: 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:03.783 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:03.784 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:03.784 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:03.784 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:03.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3729631063 0 0) 0x7f9ab41d77a0 con 0x7f9ab413f710 2024-03-25T18:41:03.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 >> v1:172.21.15.32:6790/0 conn(0x7f9ab40a98f0 legacy=0x7f9ab4138610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:03.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 >> v1:172.21.15.32:6789/0 conn(0x7f9ab41430c0 legacy=0x7f9ab41e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:03.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:41:03.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 1 monclient: found mon.b 2024-03-25T18:41:03.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ab4137870 con 0x7f9ab413f710 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9aaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 5 monclient: authenticate success, global_id 4294 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 10 monclient: _renew_subs 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ab4137b20 con 0x7f9ab413f710 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 10 client.4294.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:41:03.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 10 monclient: _renew_subs 2024-03-25T18:41:03.790 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:03.790 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ab4135220 con 0x7f9ab413f710 2024-03-25T18:41:03.790 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.788+0000 7f9abba86640 1 librados: init done 2024-03-25T18:41:03.790 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.789+0000 7f9abba86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:03.790 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.789+0000 7f9abba86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:03.790 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.789+0000 7f9abba86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:03.791 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:03.789+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ab4074cc0 con 0x7f9ab413f710 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.064+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9aa4001010 con 0x7f9ab413f710 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.064+0000 7f9aaa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.064+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9aa4004bf0 con 0x7f9ab413f710 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.064+0000 7f9aaa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.064+0000 7f9aaa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.064+0000 7f9aaa7fc640 10 monclient: dump: 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:04.065 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.064+0000 7f9aabfff640 4 set_mon_vals no callback set 2024-03-25T18:41:04.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.065+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f9aa401ac00 con 0x7f9ab413f710 2024-03-25T18:41:04.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.065+0000 7f9ab9822640 10 monclient: get_auth_request con 0x7f9a9806e060 auth_method 0 2024-03-25T18:41:04.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.066+0000 7f9aaa7fc640 10 client.4294.objecter ms_handle_connect 0x7f9a9806e060 2024-03-25T18:41:04.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.072+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f9aa40548b0 con 0x7f9ab413f710 2024-03-25T18:41:04.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.072+0000 7f9aaa7fc640 10 client.4294.objecter ms_dispatch 0x7f9ab4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:41:04.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.072+0000 7f9aaa7fc640 3 client.4294.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:41:04.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.072+0000 7f9aaa7fc640 3 client.4294.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:41:04.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.072+0000 7f9aaa7fc640 20 client.4294.objecter dump_active .. 0 homeless 2024-03-25T18:41:04.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.072+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9aa40501f0 con 0x7f9ab413f710 2024-03-25T18:41:04.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.072+0000 7f9aaa7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:04.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.072+0000 7f9aaa7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:41:04.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.312+0000 7f9abba86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:04.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.312+0000 7f9abba86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:04.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.312+0000 7f9abba86640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:04.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.312+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ab40ab9b0 con 0x7f9ab413f710 2024-03-25T18:41:04.492 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.491+0000 7f9aaa7fc640 1 -- 172.21.15.32:0/3158534371 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f9aa40522f0 con 0x7f9ab413f710 2024-03-25T18:41:04.492 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.491+0000 7f9aaa7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:04.492 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.491+0000 7f9aaa7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:41:04.492 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:41:04.493 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:41:04.494 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.493+0000 7f9abba86640 10 librados: watch_flush enter 2024-03-25T18:41:04.494 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.493+0000 7f9abba86640 10 librados: watch_flush exit 2024-03-25T18:41:04.494 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.493+0000 7f9abba86640 20 client.4294.objecter shutdown clearing up homeless session... 2024-03-25T18:41:04.494 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.493+0000 7f9abba86640 10 client.4294.objecter successfully canceled tick 2024-03-25T18:41:04.494 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 >> v1:172.21.15.136:6800/1141502808 conn(0x7f9a9806e060 legacy=0x7f9a98070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:04.494 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 10 monclient: shutdown 2024-03-25T18:41:04.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:04.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 >> v1:172.21.15.136:6789/0 conn(0x7f9ab413f710 legacy=0x7f9ab41d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:04.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 shutdown_connections 2024-03-25T18:41:04.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 >> 172.21.15.32:0/3158534371 conn(0x7f9ab40b7ea0 msgr2=0x7f9ab4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:04.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 shutdown_connections 2024-03-25T18:41:04.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 1 -- 172.21.15.32:0/3158534371 wait complete. 2024-03-25T18:41:04.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:04.494+0000 7f9abba86640 1 librados: shutdown 2024-03-25T18:41:04.505 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:41:07.283 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:19.610 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:07.286 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:19.614 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:07.320 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:19.648 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:07.343 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:19.660 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:07.351 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:19.668 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:07.360 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:19.688 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:07.365 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:19.693 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:07.382 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:19.699 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:07.401 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:19.729 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:11.313 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:41:11.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.432+0000 7f4ffad15640 10 monclient: get_monmap_and_config 2024-03-25T18:41:11.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.432+0000 7f4ffad15640 10 monclient: build_initial_monmap 2024-03-25T18:41:11.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.432+0000 7f4ffad15640 10 monclient: monmap: 2024-03-25T18:41:11.433 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:11.433 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:11.433 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:41:11.433586+0000 2024-03-25T18:41:11.433 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:41:11.433586+0000 2024-03-25T18:41:11.433 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:11.434 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:11.434 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:11.434 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:41:11.434 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:41:11.434 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:11.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.433+0000 7f4ffad15640 1 Processor -- start 2024-03-25T18:41:11.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.433+0000 7f4ffad15640 1 -- start start 2024-03-25T18:41:11.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.433+0000 7f4ffad15640 10 monclient: init 2024-03-25T18:41:11.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.433+0000 7f4ffad15640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:11.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.433+0000 7f4ffad15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:41:11.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.433+0000 7f4ffad15640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff406ab80 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:11.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.433+0000 7f4ffad15640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff406af60 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:11.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.434+0000 7f4ffad15640 10 monclient(hunting): picked mon.noname-a con 0x7f4ff40af580 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:11.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.434+0000 7f4ffad15640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff40b04d0 con 0x7f4ff40af580 2024-03-25T18:41:11.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.434+0000 7f4ffad15640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff40b1230 con 0x7f4ff406af60 2024-03-25T18:41:11.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.434+0000 7f4ffad15640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff40affb0 con 0x7f4ff406ab80 2024-03-25T18:41:11.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.434+0000 7f4ffad15640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:11.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.434+0000 7f4ffad15640 10 monclient(hunting): authenticate will time out at 6177.623570s 2024-03-25T18:41:11.443 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.442+0000 7f4ff92b2640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f4ff40af580 0x7f4ff414dc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:48862/0 (socket says 172.21.15.32:48862) 2024-03-25T18:41:11.443 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.442+0000 7f4ff92b2640 1 -- 172.21.15.32:0/1338721065 learned_addr learned my addr 172.21.15.32:0/1338721065 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:11.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.451+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1128700216 0 0) 0x7f4ff40b04d0 con 0x7f4ff40af580 2024-03-25T18:41:11.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.451+0000 7f4ff37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:41:11.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.451+0000 7f4ff37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:11.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.451+0000 7f4ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:11.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.451+0000 7f4ff37fe640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:41:11.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.451+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fe4003520 con 0x7f4ff40af580 2024-03-25T18:41:11.464 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.463+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155479704 0 0) 0x7f4ff40b1230 con 0x7f4ff406af60 2024-03-25T18:41:11.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.463+0000 7f4ff37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:41:11.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.463+0000 7f4ff37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:11.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.463+0000 7f4ff37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:11.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.463+0000 7f4ff37fe640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:41:11.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.463+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff40b04d0 con 0x7f4ff406af60 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.472+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 939615467 0 0) 0x7f4fe4003520 con 0x7f4ff40af580 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.472+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 >> v1:172.21.15.32:6790/0 conn(0x7f4ff406ab80 legacy=0x7f4ff413f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 >> v1:172.21.15.136:6789/0 conn(0x7f4ff406af60 legacy=0x7f4ff4142370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 1 monclient: found mon.noname-a 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff40b0140 con 0x7f4ff40af580 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 10 monclient: _finish_auth 0 2024-03-25T18:41:11.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4fec001a10 con 0x7f4ff40af580 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ff37fe640 10 monclient: dump: 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:11.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ffad15640 5 monclient: authenticate success, global_id 4308 2024-03-25T18:41:11.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.473+0000 7f4ffad15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:11.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.482+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fec001b70 con 0x7f4ff40af580 2024-03-25T18:41:11.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.482+0000 7f4ff37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:11.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.482+0000 7f4ff37fe640 1 -- 172.21.15.32:0/1338721065 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4fec001f80 con 0x7f4ff40af580 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.482+0000 7f4ffad15640 10 monclient: get_monmap_and_config success 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.482+0000 7f4ffad15640 4 set_mon_vals no callback set 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.482+0000 7f4ff37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.482+0000 7f4ff37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.482+0000 7f4ff37fe640 10 monclient: dump: 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:11.484 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:11.485 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:11.485 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:11.485 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:11.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.483+0000 7f4ffad15640 10 monclient: shutdown 2024-03-25T18:41:11.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.483+0000 7f4ffad15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:11.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.483+0000 7f4ffad15640 1 -- 172.21.15.32:0/1338721065 >> v1:172.21.15.32:6789/0 conn(0x7f4ff40af580 legacy=0x7f4ff414dc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:11.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.483+0000 7f4ffad15640 1 -- 172.21.15.32:0/1338721065 shutdown_connections 2024-03-25T18:41:11.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.483+0000 7f4ffad15640 1 -- 172.21.15.32:0/1338721065 >> 172.21.15.32:0/1338721065 conn(0x7f4ff40b4b80 msgr2=0x7f4ff40b4f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.483+0000 7f4ffad15640 1 -- 172.21.15.32:0/1338721065 shutdown_connections 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.483+0000 7f4ffad15640 1 -- 172.21.15.32:0/1338721065 wait complete. 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.484+0000 7f4ffad15640 10 monclient: build_initial_monmap 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.484+0000 7f4ffad15640 10 monclient: monmap: 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:11.487 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.485+0000 7f4ffad15640 1 librados: starting msgr at 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.485+0000 7f4ffad15640 1 librados: starting objecter 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.485+0000 7f4ffad15640 1 Processor -- start 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.485+0000 7f4ffad15640 1 -- start start 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.485+0000 7f4ffad15640 1 librados: setting wanted keys 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.485+0000 7f4ffad15640 1 librados: calling monclient init 2024-03-25T18:41:11.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.485+0000 7f4ffad15640 10 monclient: init 2024-03-25T18:41:11.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:11.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:41:11.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff406ab80 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:11.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff40af580 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:11.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 10 monclient(hunting): picked mon.noname-a con 0x7f4ff413be20 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:11.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff41db7e0 con 0x7f4ff413be20 2024-03-25T18:41:11.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff41dbaa0 con 0x7f4ff40af580 2024-03-25T18:41:11.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff411f680 con 0x7f4ff406ab80 2024-03-25T18:41:11.490 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:11.490 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.486+0000 7f4ffad15640 10 monclient(hunting): authenticate will time out at 6177.676110s 2024-03-25T18:41:11.492 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.492+0000 7f4ff8ab1640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f4ff406ab80 0x7f4ff41db0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:35766/0 (socket says 172.21.15.32:35766) 2024-03-25T18:41:11.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.492+0000 7f4ff8ab1640 1 -- 172.21.15.32:0/771732113 learned_addr learned my addr 172.21.15.32:0/771732113 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:11.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.493+0000 7f4ff17fa640 10 client.?.objecter ms_handle_connect 0x7f4ff406ab80 2024-03-25T18:41:11.494 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.493+0000 7f4ff17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:11.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.501+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2636923376 0 0) 0x7f4ff411f680 con 0x7f4ff406ab80 2024-03-25T18:41:11.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.501+0000 7f4ff17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:41:11.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.501+0000 7f4ff17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:11.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.501+0000 7f4ff17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:11.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.501+0000 7f4ff17fa640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:41:11.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.501+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8004eb0 con 0x7f4ff406ab80 2024-03-25T18:41:11.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.504+0000 7f4ff17fa640 10 client.?.objecter ms_handle_connect 0x7f4ff40af580 2024-03-25T18:41:11.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.504+0000 7f4ff17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 10 client.?.objecter ms_handle_connect 0x7f4ff413be20 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3224953190 0 0) 0x7f4fd8004eb0 con 0x7f4ff406ab80 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ff411f680 con 0x7f4ff406ab80 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4fe80019d0 con 0x7f4ff406ab80 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 10 monclient(hunting): dump: 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:11.513 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:11.514 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.511+0000 7f4ff17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:41:11.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.520+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013776366 0 0) 0x7f4ff41dbaa0 con 0x7f4ff40af580 2024-03-25T18:41:11.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.520+0000 7f4ff17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:41:11.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.520+0000 7f4ff17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:11.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.520+0000 7f4ff17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:11.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.520+0000 7f4ff17fa640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:41:11.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.520+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd8004eb0 con 0x7f4ff40af580 2024-03-25T18:41:11.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3866183491 0 0) 0x7f4ff411f680 con 0x7f4ff406ab80 2024-03-25T18:41:11.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 >> v1:172.21.15.136:6789/0 conn(0x7f4ff40af580 legacy=0x7f4ff413b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:11.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 >> v1:172.21.15.32:6789/0 conn(0x7f4ff413be20 legacy=0x7f4ff411d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:11.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:41:11.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 1 monclient: found mon.c 2024-03-25T18:41:11.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:11.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff413b820 con 0x7f4ff406ab80 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 10 monclient: _finish_auth 0 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ff17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ffad15640 5 monclient: authenticate success, global_id 4304 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ffad15640 10 monclient: _renew_subs 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ffad15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.538+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff413ba10 con 0x7f4ff406ab80 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.539+0000 7f4ffad15640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.539+0000 7f4ffad15640 10 monclient: _renew_subs 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.539+0000 7f4ffad15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.539+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff411d910 con 0x7f4ff406ab80 2024-03-25T18:41:11.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.539+0000 7f4ffad15640 1 librados: init done 2024-03-25T18:41:11.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.540+0000 7f4ffad15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:11.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.540+0000 7f4ffad15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:11.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.540+0000 7f4ffad15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:41:11.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.540+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff411f480 con 0x7f4ff406ab80 2024-03-25T18:41:11.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.880+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe8001b30 con 0x7f4ff406ab80 2024-03-25T18:41:12.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.880+0000 7f4ff17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:12.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.880+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4fe8003a60 con 0x7f4ff406ab80 2024-03-25T18:41:12.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.880+0000 7f4ff17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:12.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.880+0000 7f4ff17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:41:12.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.880+0000 7f4ff17fa640 10 monclient: dump: 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:12.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.880+0000 7f4ff2ffd640 4 set_mon_vals no callback set 2024-03-25T18:41:12.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.881+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f4fe8003c80 con 0x7f4ff406ab80 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.882+0000 7f4ff3fff640 10 monclient: get_auth_request con 0x7f4fd806ddd0 auth_method 0 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.882+0000 7f4ff17fa640 10 client.4304.objecter ms_handle_connect 0x7f4fd806ddd0 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.890+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f4fe8054850 con 0x7f4ff406ab80 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.890+0000 7f4ff17fa640 10 client.4304.objecter ms_dispatch 0x7f4ff4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.890+0000 7f4ff17fa640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.890+0000 7f4ff17fa640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.890+0000 7f4ff17fa640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.890+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fe8050360 con 0x7f4ff406ab80 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.890+0000 7f4ff17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:12.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:11.890+0000 7f4ff17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:41:12.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.149+0000 7f4ffad15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:12.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.149+0000 7f4ffad15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:12.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.149+0000 7f4ffad15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:41:12.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.149+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ff40627b0 con 0x7f4ff406ab80 2024-03-25T18:41:12.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.174+0000 7f4ff17fa640 1 -- 172.21.15.32:0/771732113 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f4fe8052290 con 0x7f4ff406ab80 2024-03-25T18:41:12.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.174+0000 7f4ff17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:12.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.174+0000 7f4ff17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:41:12.175 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:41:12.176 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 10 librados: watch_flush enter 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 10 librados: watch_flush exit 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 10 client.4304.objecter successfully canceled tick 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 >> v1:172.21.15.136:6800/1141502808 conn(0x7f4fd806ddd0 legacy=0x7f4fd8070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 10 monclient: shutdown 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 >> v1:172.21.15.32:6790/0 conn(0x7f4ff406ab80 legacy=0x7f4ff41db0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 shutdown_connections 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 >> 172.21.15.32:0/771732113 conn(0x7f4ff40b4b80 msgr2=0x7f4ff40b1cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 shutdown_connections 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 1 -- 172.21.15.32:0/771732113 wait complete. 2024-03-25T18:41:12.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:12.176+0000 7f4ffad15640 1 librados: shutdown 2024-03-25T18:41:12.189 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:41:18.997 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:41:19.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.107+0000 7fd86bdd0640 10 monclient: get_monmap_and_config 2024-03-25T18:41:19.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.107+0000 7fd86bdd0640 10 monclient: build_initial_monmap 2024-03-25T18:41:19.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.107+0000 7fd86bdd0640 10 monclient: monmap: 2024-03-25T18:41:19.108 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:41:19.108736+0000 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:41:19.108736+0000 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:41:19.109 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:19.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.108+0000 7fd86bdd0640 1 Processor -- start 2024-03-25T18:41:19.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.108+0000 7fd86bdd0640 1 -- start start 2024-03-25T18:41:19.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.108+0000 7fd86bdd0640 10 monclient: init 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 10 monclient(hunting): picked mon.noname-c con 0x7fd8640a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 10 monclient(hunting): picked mon.noname-b con 0x7fd86413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 10 monclient(hunting): picked mon.noname-a con 0x7fd8641430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd864142520 con 0x7fd8641430c0 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640b28a0 con 0x7fd8640a98f0 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640b2a30 con 0x7fd86413f710 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:19.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.109+0000 7fd86bdd0640 10 monclient(hunting): authenticate will time out at 6185.298831s 2024-03-25T18:41:19.116 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.115+0000 7fd86936b640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fd86413f710 0x7fd8641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:47946/0 (socket says 172.21.15.32:47946) 2024-03-25T18:41:19.116 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.115+0000 7fd86936b640 1 -- 172.21.15.32:0/3610652568 learned_addr learned my addr 172.21.15.32:0/3610652568 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:19.127 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2632886581 0 0) 0x7fd864142520 con 0x7fd8641430c0 2024-03-25T18:41:19.127 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 10 monclient(hunting): _negotiate 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd850003520 con 0x7fd8641430c0 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2900699974 0 0) 0x7fd8640b2a30 con 0x7fd86413f710 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 10 monclient(hunting): _negotiate 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:41:19.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.126+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd864142520 con 0x7fd86413f710 2024-03-25T18:41:19.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.134+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3227669480 0 0) 0x7fd8640b28a0 con 0x7fd8640a98f0 2024-03-25T18:41:19.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.134+0000 7fd868b6a640 10 monclient(hunting): _negotiate 2024-03-25T18:41:19.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.134+0000 7fd868b6a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:19.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.134+0000 7fd868b6a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:19.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.134+0000 7fd868b6a640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:41:19.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.134+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8640b2a30 con 0x7fd8640a98f0 2024-03-25T18:41:19.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1748949335 0 0) 0x7fd864142520 con 0x7fd86413f710 2024-03-25T18:41:19.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 >> v1:172.21.15.136:6789/0 conn(0x7fd8640a98f0 legacy=0x7fd8640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:19.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 >> v1:172.21.15.32:6789/0 conn(0x7fd8641430c0 legacy=0x7fd86414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:19.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 10 monclient: _finish_hunting 0 2024-03-25T18:41:19.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 1 monclient: found mon.noname-b 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8640b44c0 con 0x7fd86413f710 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 10 monclient: _finish_auth 0 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 10 monclient: _check_auth_tickets 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd854001670 con 0x7fd86413f710 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd868b6a640 10 monclient: dump: 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.156 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:19.157 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:19.157 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:19.157 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:19.157 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:19.157 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:19.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd86bdd0640 5 monclient: authenticate success, global_id 4307 2024-03-25T18:41:19.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.154+0000 7fd86bdd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:19.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.180+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8540017d0 con 0x7fd86413f710 2024-03-25T18:41:19.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.180+0000 7fd868b6a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:19.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.180+0000 7fd868b6a640 1 -- 172.21.15.32:0/3610652568 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd854001be0 con 0x7fd86413f710 2024-03-25T18:41:19.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.180+0000 7fd868b6a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.180+0000 7fd868b6a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.180+0000 7fd868b6a640 10 monclient: dump: 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.180+0000 7fd86bdd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:41:19.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.180+0000 7fd86bdd0640 4 set_mon_vals no callback set 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.181+0000 7fd86bdd0640 10 monclient: shutdown 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.181+0000 7fd86bdd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.181+0000 7fd86bdd0640 1 -- 172.21.15.32:0/3610652568 >> v1:172.21.15.32:6790/0 conn(0x7fd86413f710 legacy=0x7fd8641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.181+0000 7fd86bdd0640 1 -- 172.21.15.32:0/3610652568 shutdown_connections 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.181+0000 7fd86bdd0640 1 -- 172.21.15.32:0/3610652568 >> 172.21.15.32:0/3610652568 conn(0x7fd8640b7ea0 msgr2=0x7fd8640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.181+0000 7fd86bdd0640 1 -- 172.21.15.32:0/3610652568 shutdown_connections 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.181+0000 7fd86bdd0640 1 -- 172.21.15.32:0/3610652568 wait complete. 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.182+0000 7fd86bdd0640 10 monclient: build_initial_monmap 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.182+0000 7fd86bdd0640 10 monclient: monmap: 2024-03-25T18:41:19.184 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 1 librados: starting msgr at 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 1 librados: starting objecter 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 1 Processor -- start 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 1 -- start start 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 1 librados: setting wanted keys 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 1 librados: calling monclient init 2024-03-25T18:41:19.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 10 monclient: init 2024-03-25T18:41:19.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 10 monclient(hunting): picked mon.noname-a con 0x7fd8640a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.183+0000 7fd86bdd0640 10 monclient(hunting): picked mon.noname-b con 0x7fd86413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.184+0000 7fd86bdd0640 10 monclient(hunting): picked mon.noname-c con 0x7fd8641430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.184+0000 7fd86bdd0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd864138ed0 con 0x7fd8640a98f0 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.184+0000 7fd86bdd0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640abc80 con 0x7fd86413f710 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.184+0000 7fd86bdd0640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8640abf60 con 0x7fd8641430c0 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.184+0000 7fd86bdd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:19.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.184+0000 7fd86bdd0640 10 monclient(hunting): authenticate will time out at 6185.373668s 2024-03-25T18:41:19.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.198+0000 7fd86a36d640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fd8641430c0 0x7fd8641e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:47990/0 (socket says 172.21.15.32:47990) 2024-03-25T18:41:19.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.198+0000 7fd86a36d640 1 -- 172.21.15.32:0/1527179034 learned_addr learned my addr 172.21.15.32:0/1527179034 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:19.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.200+0000 7fd85a7fc640 10 client.?.objecter ms_handle_connect 0x7fd8641430c0 2024-03-25T18:41:19.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.200+0000 7fd85a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:19.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.203+0000 7fd85a7fc640 10 client.?.objecter ms_handle_connect 0x7fd8640a98f0 2024-03-25T18:41:19.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.203+0000 7fd85a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:19.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.206+0000 7fd85a7fc640 10 client.?.objecter ms_handle_connect 0x7fd86413f710 2024-03-25T18:41:19.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.206+0000 7fd85a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:19.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.211+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1061570735 0 0) 0x7fd8640abc80 con 0x7fd86413f710 2024-03-25T18:41:19.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.211+0000 7fd85a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:19.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.211+0000 7fd85a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:19.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.211+0000 7fd85a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:19.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.211+0000 7fd85a7fc640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:41:19.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.211+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd848004eb0 con 0x7fd86413f710 2024-03-25T18:41:19.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.219+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3857716273 0 0) 0x7fd8640abf60 con 0x7fd8641430c0 2024-03-25T18:41:19.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.219+0000 7fd85a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:19.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.219+0000 7fd85a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:19.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.219+0000 7fd85a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:19.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.219+0000 7fd85a7fc640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:41:19.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.219+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8640abc80 con 0x7fd8641430c0 2024-03-25T18:41:19.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.221+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 933008216 0 0) 0x7fd864138ed0 con 0x7fd8640a98f0 2024-03-25T18:41:19.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.221+0000 7fd85a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:19.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.221+0000 7fd85a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:19.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.221+0000 7fd85a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:19.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.221+0000 7fd85a7fc640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:41:19.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.221+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8640abf60 con 0x7fd8640a98f0 2024-03-25T18:41:19.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.230+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3387655284 0 0) 0x7fd8640abc80 con 0x7fd8641430c0 2024-03-25T18:41:19.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.230+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd864138ed0 con 0x7fd8641430c0 2024-03-25T18:41:19.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.230+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd85c001bf0 con 0x7fd8641430c0 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.230+0000 7fd85a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.230+0000 7fd85a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.230+0000 7fd85a7fc640 10 monclient(hunting): dump: 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:19.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.230+0000 7fd85a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:41:19.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.231+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3592315627 0 0) 0x7fd848004eb0 con 0x7fd86413f710 2024-03-25T18:41:19.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.231+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8640abc80 con 0x7fd86413f710 2024-03-25T18:41:19.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.231+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd854001ba0 con 0x7fd86413f710 2024-03-25T18:41:19.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.231+0000 7fd85a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:19.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.231+0000 7fd85a7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.231+0000 7fd85a7fc640 10 monclient(hunting): dump: 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:19.234 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 384622141 0 0) 0x7fd8640abc80 con 0x7fd86413f710 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.32:6790/0 conn(0x7fd8641430c0 legacy=0x7fd8641e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.32:6789/0 conn(0x7fd8640a98f0 legacy=0x7fd8641387c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 1 monclient: found mon.b 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd864139170 con 0x7fd86413f710 2024-03-25T18:41:19.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd85a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.244+0000 7fd86bdd0640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.245+0000 7fd86bdd0640 10 monclient: _renew_subs 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.245+0000 7fd86bdd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.245+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8640acd70 con 0x7fd86413f710 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.245+0000 7fd86bdd0640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.245+0000 7fd86bdd0640 10 monclient: _renew_subs 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.245+0000 7fd86bdd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.245+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd864136c30 con 0x7fd86413f710 2024-03-25T18:41:19.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.245+0000 7fd86bdd0640 1 librados: init done 2024-03-25T18:41:19.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.246+0000 7fd86bdd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:19.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.246+0000 7fd86bdd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:19.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.246+0000 7fd86bdd0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:19.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.246+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd864136690 con 0x7fd86413f710 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd86936b640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.136:6789/0 conn(0x7fd86413f710 legacy=0x7fd8640ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd86936b640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.136:6789/0 conn(0x7fd86413f710 legacy=0x7fd8640ac660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd86936b640 1 --1- 172.21.15.32:0/1527179034 >> v1:172.21.15.136:6789/0 conn(0x7fd86413f710 0x7fd8640ac660 :-1 s=OPENED pgs=11328 cs=1 l=1).handle_message read tag failed 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd86936b640 1 --1- 172.21.15.32:0/1527179034 >> v1:172.21.15.136:6789/0 conn(0x7fd86413f710 0x7fd8640ac660 :-1 s=OPENED pgs=11328 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd854004a90 con 0x7fd86413f710 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 10 monclient: ms_handle_reset current mon v1:172.21.15.136:6789/0 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.136:6789/0 conn(0x7fd86413f710 legacy=0x7fd8640ac660 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 10 monclient(hunting): picked mon.b con 0x7fd8480066b0 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85bfff640 4 set_mon_vals no callback set 2024-03-25T18:41:19.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 10 monclient(hunting): picked mon.a con 0x7fd8640a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:19.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 10 monclient(hunting): picked mon.c con 0x7fd8641430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:19.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd8640abc80 con 0x7fd8640a98f0 2024-03-25T18:41:19.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd848004eb0 con 0x7fd8480066b0 2024-03-25T18:41:19.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7fd848013d30 con 0x7fd8641430c0 2024-03-25T18:41:19.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.499+0000 7fd85a7fc640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:19.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.512+0000 7fd85a7fc640 10 client.4306.objecter ms_handle_connect 0x7fd8641430c0 2024-03-25T18:41:19.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.512+0000 7fd85a7fc640 10 client.4306.objecter resend_mon_ops 2024-03-25T18:41:19.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.512+0000 7fd85a7fc640 10 client.4306.objecter ms_handle_connect 0x7fd8640a98f0 2024-03-25T18:41:19.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.512+0000 7fd85a7fc640 10 client.4306.objecter resend_mon_ops 2024-03-25T18:41:19.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.519+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2085877184 0 0) 0x7fd848013d30 con 0x7fd8641430c0 2024-03-25T18:41:19.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.519+0000 7fd85a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:19.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.519+0000 7fd85a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:19.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.519+0000 7fd85a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:41:19.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.519+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fd848011c30 con 0x7fd8641430c0 2024-03-25T18:41:19.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.528+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3311038194 0 0) 0x7fd8640abc80 con 0x7fd8640a98f0 2024-03-25T18:41:19.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.528+0000 7fd85a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:19.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.528+0000 7fd85a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:19.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.528+0000 7fd85a7fc640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-25T18:41:19.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.528+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7fd848013d30 con 0x7fd8640a98f0 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 1990360504 0 0) 0x7fd848011c30 con 0x7fd8641430c0 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.136:6789/0 conn(0x7fd8480066b0 legacy=0x7fd8480092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.32:6789/0 conn(0x7fd8640a98f0 legacy=0x7fd84800ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 1 monclient: found mon.c 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7fd848015110 con 0x7fd8641430c0 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:41:19.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd848004eb0 con 0x7fd8641430c0 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd85c003990 con 0x7fd8641430c0 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.529+0000 7fd85a7fc640 10 monclient: dump: 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:19.532 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:19.533 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:19.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.853+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd85c003bb0 con 0x7fd8641430c0 2024-03-25T18:41:19.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.853+0000 7fd85a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:19.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.853+0000 7fd85b7fe640 4 set_mon_vals no callback set 2024-03-25T18:41:19.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.853+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fd85c003da0 con 0x7fd8641430c0 2024-03-25T18:41:19.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.854+0000 7fd869b6c640 10 monclient: get_auth_request con 0x7fd848079a40 auth_method 0 2024-03-25T18:41:19.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.854+0000 7fd85a7fc640 10 client.4306.objecter ms_handle_connect 0x7fd848079a40 2024-03-25T18:41:19.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.860+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fd85c054920 con 0x7fd8641430c0 2024-03-25T18:41:19.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.860+0000 7fd85a7fc640 10 client.4306.objecter ms_dispatch 0x7fd864000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:41:19.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.860+0000 7fd85a7fc640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:41:19.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.860+0000 7fd85a7fc640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:41:19.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.860+0000 7fd85a7fc640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T18:41:19.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.860+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd85c050430 con 0x7fd8641430c0 2024-03-25T18:41:19.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.860+0000 7fd85a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:19.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:19.860+0000 7fd85a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:41:20.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.112+0000 7fd86bdd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:20.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.112+0000 7fd86bdd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:20.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.112+0000 7fd86bdd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:41:20.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.112+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8640627b0 con 0x7fd8641430c0 2024-03-25T18:41:20.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.145+0000 7fd85a7fc640 1 -- 172.21.15.32:0/1527179034 <== mon.2 v1:172.21.15.32:6790/0 8 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fd85c052360 con 0x7fd8641430c0 2024-03-25T18:41:20.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.145+0000 7fd85a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:20.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.145+0000 7fd85a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:41:20.146 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:41:20.146 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.147+0000 7fd86bdd0640 10 librados: watch_flush enter 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.147+0000 7fd86bdd0640 10 librados: watch_flush exit 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.147+0000 7fd86bdd0640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.147+0000 7fd86bdd0640 10 client.4306.objecter successfully canceled tick 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.147+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.136:6800/1141502808 conn(0x7fd848079a40 legacy=0x7fd84807bf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.147+0000 7fd86bdd0640 10 monclient: shutdown 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.147+0000 7fd86bdd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.147+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 >> v1:172.21.15.32:6790/0 conn(0x7fd8641430c0 legacy=0x7fd848010150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:20.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.148+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 shutdown_connections 2024-03-25T18:41:20.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.148+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 >> 172.21.15.32:0/1527179034 conn(0x7fd8640b7ea0 msgr2=0x7fd864074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:20.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.148+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 shutdown_connections 2024-03-25T18:41:20.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.148+0000 7fd86bdd0640 1 -- 172.21.15.32:0/1527179034 wait complete. 2024-03-25T18:41:20.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:20.148+0000 7fd86bdd0640 1 librados: shutdown 2024-03-25T18:41:20.160 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:41:26.968 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:41:27.083 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.082+0000 7fd55aa3e640 10 monclient: get_monmap_and_config 2024-03-25T18:41:27.083 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.082+0000 7fd55aa3e640 10 monclient: build_initial_monmap 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.082+0000 7fd55aa3e640 10 monclient: monmap: 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:41:27.084038+0000 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:41:27.084038+0000 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:41:27.084 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:27.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.083+0000 7fd55aa3e640 1 Processor -- start 2024-03-25T18:41:27.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.083+0000 7fd55aa3e640 1 -- start start 2024-03-25T18:41:27.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.083+0000 7fd55aa3e640 10 monclient: init 2024-03-25T18:41:27.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:27.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:41:27.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 10 monclient(hunting): picked mon.noname-a con 0x7fd5540a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:27.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 10 monclient(hunting): picked mon.noname-c con 0x7fd55413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:27.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 10 monclient(hunting): picked mon.noname-b con 0x7fd5541430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:27.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd554142520 con 0x7fd5540a98f0 2024-03-25T18:41:27.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5540b28a0 con 0x7fd55413f710 2024-03-25T18:41:27.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5540b2a30 con 0x7fd5541430c0 2024-03-25T18:41:27.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:27.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.084+0000 7fd55aa3e640 10 monclient(hunting): authenticate will time out at 6193.273666s 2024-03-25T18:41:27.090 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.089+0000 7fd5537fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd55413f710 0x7fd5541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:55746/0 (socket says 172.21.15.32:55746) 2024-03-25T18:41:27.090 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.089+0000 7fd5537fe640 1 -- 172.21.15.32:0/171643757 learned_addr learned my addr 172.21.15.32:0/171643757 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:27.104 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.103+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484612463 0 0) 0x7fd5540b28a0 con 0x7fd55413f710 2024-03-25T18:41:27.104 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.103+0000 7fd552ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:41:27.104 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.103+0000 7fd552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:27.104 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.103+0000 7fd552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:27.104 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.103+0000 7fd552ffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:41:27.104 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.103+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd53c003520 con 0x7fd55413f710 2024-03-25T18:41:27.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.114+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3217016442 0 0) 0x7fd5540b2a30 con 0x7fd5541430c0 2024-03-25T18:41:27.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.114+0000 7fd552ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:41:27.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.114+0000 7fd552ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:27.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.114+0000 7fd552ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:27.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.114+0000 7fd552ffd640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:41:27.116 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.114+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5540b28a0 con 0x7fd5541430c0 2024-03-25T18:41:27.116 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 185721266 0 0) 0x7fd53c003520 con 0x7fd55413f710 2024-03-25T18:41:27.116 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 >> v1:172.21.15.32:6790/0 conn(0x7fd5541430c0 legacy=0x7fd55414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 >> v1:172.21.15.32:6789/0 conn(0x7fd5540a98f0 legacy=0x7fd5540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 1 monclient: found mon.noname-c 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5540b44c0 con 0x7fd55413f710 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 10 monclient: _finish_auth 0 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd548001690 con 0x7fd55413f710 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd552ffd640 10 monclient: dump: 2024-03-25T18:41:27.117 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd55aa3e640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:41:27.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.115+0000 7fd55aa3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:27.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5480017f0 con 0x7fd55413f710 2024-03-25T18:41:27.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd552ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:27.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd552ffd640 1 -- 172.21.15.32:0/171643757 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd548001c00 con 0x7fd55413f710 2024-03-25T18:41:27.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd552ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:27.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd552ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:41:27.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd552ffd640 10 monclient: dump: 2024-03-25T18:41:27.141 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:27.141 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd55aa3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:41:27.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd55aa3e640 4 set_mon_vals no callback set 2024-03-25T18:41:27.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd55aa3e640 10 monclient: shutdown 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd55aa3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.140+0000 7fd55aa3e640 1 -- 172.21.15.32:0/171643757 >> v1:172.21.15.136:6789/0 conn(0x7fd55413f710 legacy=0x7fd5541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.141+0000 7fd55aa3e640 1 -- 172.21.15.32:0/171643757 shutdown_connections 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.141+0000 7fd55aa3e640 1 -- 172.21.15.32:0/171643757 >> 172.21.15.32:0/171643757 conn(0x7fd5540b7ea0 msgr2=0x7fd5540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.141+0000 7fd55aa3e640 1 -- 172.21.15.32:0/171643757 shutdown_connections 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.141+0000 7fd55aa3e640 1 -- 172.21.15.32:0/171643757 wait complete. 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 10 monclient: build_initial_monmap 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 10 monclient: monmap: 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:27.145 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 1 librados: starting msgr at 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 1 librados: starting objecter 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 1 Processor -- start 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 1 -- start start 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 1 librados: setting wanted keys 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 1 librados: calling monclient init 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.142+0000 7fd55aa3e640 10 monclient: init 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:41:27.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 10 monclient(hunting): picked mon.noname-c con 0x7fd5540a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:27.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 10 monclient(hunting): picked mon.noname-b con 0x7fd55413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:27.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 10 monclient(hunting): picked mon.noname-a con 0x7fd5541430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:27.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd554138c10 con 0x7fd5541430c0 2024-03-25T18:41:27.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd554138e60 con 0x7fd55413f710 2024-03-25T18:41:27.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd554139140 con 0x7fd5540a98f0 2024-03-25T18:41:27.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:27.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.143+0000 7fd55aa3e640 10 monclient(hunting): authenticate will time out at 6193.332984s 2024-03-25T18:41:27.158 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.157+0000 7fd5537fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd55413f710 0x7fd554132440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:55810/0 (socket says 172.21.15.32:55810) 2024-03-25T18:41:27.158 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.157+0000 7fd5537fe640 1 -- 172.21.15.32:0/1620701121 learned_addr learned my addr 172.21.15.32:0/1620701121 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:27.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.160+0000 7fd550ff9640 10 client.?.objecter ms_handle_connect 0x7fd55413f710 2024-03-25T18:41:27.161 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.160+0000 7fd550ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:27.165 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.164+0000 7fd550ff9640 10 client.?.objecter ms_handle_connect 0x7fd5540a98f0 2024-03-25T18:41:27.165 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.164+0000 7fd550ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:27.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.165+0000 7fd550ff9640 10 client.?.objecter ms_handle_connect 0x7fd5541430c0 2024-03-25T18:41:27.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.165+0000 7fd550ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:27.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.169+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 803381267 0 0) 0x7fd554138e60 con 0x7fd55413f710 2024-03-25T18:41:27.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.169+0000 7fd550ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:41:27.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.169+0000 7fd550ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:27.171 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.169+0000 7fd550ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:27.171 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.169+0000 7fd550ff9640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:41:27.171 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.169+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd540004eb0 con 0x7fd55413f710 2024-03-25T18:41:27.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.180+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342222663 0 0) 0x7fd554138c10 con 0x7fd5541430c0 2024-03-25T18:41:27.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.180+0000 7fd550ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:41:27.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.180+0000 7fd550ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:27.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.180+0000 7fd550ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:27.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.180+0000 7fd550ff9640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:41:27.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.180+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd554138e60 con 0x7fd5541430c0 2024-03-25T18:41:27.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.182+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 309585332 0 0) 0x7fd540004eb0 con 0x7fd55413f710 2024-03-25T18:41:27.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.182+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd554138c10 con 0x7fd55413f710 2024-03-25T18:41:27.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.182+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd548001c00 con 0x7fd55413f710 2024-03-25T18:41:27.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.182+0000 7fd550ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.182+0000 7fd550ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.182+0000 7fd550ff9640 10 monclient(hunting): dump: 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:27.184 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:27.185 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:27.185 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:27.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.182+0000 7fd550ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:41:27.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.184+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274644664 0 0) 0x7fd554139140 con 0x7fd5540a98f0 2024-03-25T18:41:27.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.184+0000 7fd550ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:41:27.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.184+0000 7fd550ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:27.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.184+0000 7fd550ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:27.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.184+0000 7fd550ff9640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:41:27.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.184+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd540004eb0 con 0x7fd5540a98f0 2024-03-25T18:41:27.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.190+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 23172340 0 0) 0x7fd554138e60 con 0x7fd5541430c0 2024-03-25T18:41:27.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.190+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd554139140 con 0x7fd5541430c0 2024-03-25T18:41:27.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.190+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd54c001b90 con 0x7fd5541430c0 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.190+0000 7fd550ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.190+0000 7fd550ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.190+0000 7fd550ff9640 10 monclient(hunting): dump: 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:27.192 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:27.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2761097462 0 0) 0x7fd554138c10 con 0x7fd55413f710 2024-03-25T18:41:27.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 >> v1:172.21.15.32:6790/0 conn(0x7fd5540a98f0 legacy=0x7fd554137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 >> v1:172.21.15.32:6789/0 conn(0x7fd5541430c0 legacy=0x7fd5541e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 1 monclient: found mon.b 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd554139920 con 0x7fd55413f710 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 10 monclient: _finish_auth 0 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:41:27.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd550ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 10 monclient: _renew_subs 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5541382e0 con 0x7fd55413f710 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 10 monclient: _renew_subs 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5540ab8b0 con 0x7fd55413f710 2024-03-25T18:41:27.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.198+0000 7fd55aa3e640 1 librados: init done 2024-03-25T18:41:27.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.199+0000 7fd55aa3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:27.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.199+0000 7fd55aa3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:27.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.199+0000 7fd55aa3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:27.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.199+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd554137a20 con 0x7fd55413f710 2024-03-25T18:41:27.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.459+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd548004a90 con 0x7fd55413f710 2024-03-25T18:41:27.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.459+0000 7fd550ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:27.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.459+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd548004e20 con 0x7fd55413f710 2024-03-25T18:41:27.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.459+0000 7fd550ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:27.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.459+0000 7fd550ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:41:27.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.459+0000 7fd550ff9640 10 monclient: dump: 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:27.461 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.459+0000 7fd5527fc640 4 set_mon_vals no callback set 2024-03-25T18:41:27.462 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.459+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fd548005040 con 0x7fd55413f710 2024-03-25T18:41:27.462 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.460+0000 7fd553fff640 10 monclient: get_auth_request con 0x7fd54006dfb0 auth_method 0 2024-03-25T18:41:27.462 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.460+0000 7fd550ff9640 10 client.4312.objecter ms_handle_connect 0x7fd54006dfb0 2024-03-25T18:41:27.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.465+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fd548054960 con 0x7fd55413f710 2024-03-25T18:41:27.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.465+0000 7fd550ff9640 10 client.4312.objecter ms_dispatch 0x7fd554000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:41:27.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.465+0000 7fd550ff9640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:41:27.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.465+0000 7fd550ff9640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:41:27.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.465+0000 7fd550ff9640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T18:41:27.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.465+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd548050470 con 0x7fd55413f710 2024-03-25T18:41:27.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.465+0000 7fd550ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:27.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.465+0000 7fd550ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:41:27.543 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:39.871 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.556 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:39.884 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.585 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:39.913 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.598 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:39.915 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.601 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:39.918 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.623 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:39.951 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.629 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:39.946 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.639 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:39.967 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.676 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:40.004 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:27.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.685+0000 7fd55aa3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:27.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.685+0000 7fd55aa3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:27.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.685+0000 7fd55aa3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:27.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.685+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd55413d010 con 0x7fd55413f710 2024-03-25T18:41:27.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.741+0000 7fd550ff9640 1 -- 172.21.15.32:0/1620701121 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fd5480523a0 con 0x7fd55413f710 2024-03-25T18:41:27.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.741+0000 7fd550ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:27.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.741+0000 7fd550ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:41:27.743 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:41:27.743 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:41:27.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 10 librados: watch_flush enter 2024-03-25T18:41:27.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 10 librados: watch_flush exit 2024-03-25T18:41:27.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T18:41:27.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 10 client.4312.objecter successfully canceled tick 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 >> v1:172.21.15.136:6800/1141502808 conn(0x7fd54006dfb0 legacy=0x7fd540070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 10 monclient: shutdown 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 >> v1:172.21.15.136:6789/0 conn(0x7fd55413f710 legacy=0x7fd554132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 shutdown_connections 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 >> 172.21.15.32:0/1620701121 conn(0x7fd5540b7ea0 msgr2=0x7fd554074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 shutdown_connections 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 1 -- 172.21.15.32:0/1620701121 wait complete. 2024-03-25T18:41:27.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:27.744+0000 7fd55aa3e640 1 librados: shutdown 2024-03-25T18:41:27.759 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:41:34.567 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:41:34.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.663+0000 7f5e09863640 10 monclient: get_monmap_and_config 2024-03-25T18:41:34.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.663+0000 7f5e09863640 10 monclient: build_initial_monmap 2024-03-25T18:41:34.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.663+0000 7f5e09863640 10 monclient: monmap: 2024-03-25T18:41:34.664 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:34.664 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:34.664 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:41:34.664147+0000 2024-03-25T18:41:34.664 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:41:34.664147+0000 2024-03-25T18:41:34.664 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:34.665 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:34.665 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:34.665 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:41:34.665 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:41:34.665 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:34.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 1 Processor -- start 2024-03-25T18:41:34.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 1 -- start start 2024-03-25T18:41:34.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 10 monclient: init 2024-03-25T18:41:34.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:34.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:41:34.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 10 monclient(hunting): picked mon.noname-a con 0x7f5e040a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:34.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 10 monclient(hunting): picked mon.noname-c con 0x7f5e0413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:34.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 10 monclient(hunting): picked mon.noname-b con 0x7f5e041430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:34.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.664+0000 7f5e09863640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e04142520 con 0x7f5e040a98f0 2024-03-25T18:41:34.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.665+0000 7f5e09863640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e040b28a0 con 0x7f5e0413f710 2024-03-25T18:41:34.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.665+0000 7f5e09863640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e040b2a30 con 0x7f5e041430c0 2024-03-25T18:41:34.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.665+0000 7f5e09863640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:34.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.665+0000 7f5e09863640 10 monclient(hunting): authenticate will time out at 6200.853581s 2024-03-25T18:41:34.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.674+0000 7f5e02ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5e040a98f0 0x7f5e040a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51250/0 (socket says 172.21.15.32:51250) 2024-03-25T18:41:34.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.674+0000 7f5e02ffd640 1 -- 172.21.15.32:0/501818567 learned_addr learned my addr 172.21.15.32:0/501818567 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:34.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.688+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285386216 0 0) 0x7f5e040b28a0 con 0x7f5e0413f710 2024-03-25T18:41:34.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.688+0000 7f5e01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:41:34.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.688+0000 7f5e01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:34.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.688+0000 7f5e01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:34.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.688+0000 7f5e01ffb640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:41:34.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.688+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5dec003520 con 0x7f5e0413f710 2024-03-25T18:41:34.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.690+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 820590982 0 0) 0x7f5e04142520 con 0x7f5e040a98f0 2024-03-25T18:41:34.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.690+0000 7f5e01ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:41:34.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.690+0000 7f5e01ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:34.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.690+0000 7f5e01ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:34.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.690+0000 7f5e01ffb640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:41:34.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.690+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e040b28a0 con 0x7f5e040a98f0 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2025512138 0 0) 0x7f5dec003520 con 0x7f5e0413f710 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 >> v1:172.21.15.32:6790/0 conn(0x7f5e041430c0 legacy=0x7f5e0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 >> v1:172.21.15.32:6789/0 conn(0x7f5e040a98f0 legacy=0x7f5e040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 1 monclient: found mon.noname-c 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e040b44c0 con 0x7f5e0413f710 2024-03-25T18:41:34.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 10 monclient: _finish_auth 0 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5df8001690 con 0x7f5e0413f710 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e01ffb640 10 monclient: dump: 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:34.702 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:34.703 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:34.703 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:34.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e09863640 5 monclient: authenticate success, global_id 4315 2024-03-25T18:41:34.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.701+0000 7f5e09863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:34.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5df80017f0 con 0x7f5e0413f710 2024-03-25T18:41:34.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e01ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:34.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e01ffb640 1 -- 172.21.15.32:0/501818567 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5df8001c00 con 0x7f5e0413f710 2024-03-25T18:41:34.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e09863640 10 monclient: get_monmap_and_config success 2024-03-25T18:41:34.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e09863640 4 set_mon_vals no callback set 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e01ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e01ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e01ffb640 10 monclient: dump: 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:34.707 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:34.708 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:34.708 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:34.708 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:34.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e09863640 10 monclient: shutdown 2024-03-25T18:41:34.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e09863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:34.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.706+0000 7f5e09863640 1 -- 172.21.15.32:0/501818567 >> v1:172.21.15.136:6789/0 conn(0x7f5e0413f710 legacy=0x7f5e041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:34.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.707+0000 7f5e09863640 1 -- 172.21.15.32:0/501818567 shutdown_connections 2024-03-25T18:41:34.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.707+0000 7f5e09863640 1 -- 172.21.15.32:0/501818567 >> 172.21.15.32:0/501818567 conn(0x7f5e040b7ea0 msgr2=0x7f5e040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:34.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.707+0000 7f5e09863640 1 -- 172.21.15.32:0/501818567 shutdown_connections 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.707+0000 7f5e09863640 1 -- 172.21.15.32:0/501818567 wait complete. 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.707+0000 7f5e09863640 10 monclient: build_initial_monmap 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.707+0000 7f5e09863640 10 monclient: monmap: 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:41:34.711 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 1 librados: starting msgr at 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 1 librados: starting objecter 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 1 Processor -- start 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 1 -- start start 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 1 librados: setting wanted keys 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 1 librados: calling monclient init 2024-03-25T18:41:34.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 10 monclient: init 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 10 monclient(hunting): picked mon.noname-a con 0x7f5e040a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 10 monclient(hunting): picked mon.noname-b con 0x7f5e0413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.708+0000 7f5e09863640 10 monclient(hunting): picked mon.noname-c con 0x7f5e041430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.709+0000 7f5e09863640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e04138a50 con 0x7f5e040a98f0 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.709+0000 7f5e09863640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e040ab9e0 con 0x7f5e0413f710 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.709+0000 7f5e09863640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e041d7880 con 0x7f5e041430c0 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.709+0000 7f5e09863640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:34.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.709+0000 7f5e09863640 10 monclient(hunting): authenticate will time out at 6200.897584s 2024-03-25T18:41:34.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.717+0000 7f5e027fc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5e0413f710 0x7f5e041d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:60246/0 (socket says 172.21.15.32:60246) 2024-03-25T18:41:34.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.718+0000 7f5e027fc640 1 -- 172.21.15.32:0/479872967 learned_addr learned my addr 172.21.15.32:0/479872967 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:34.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.718+0000 7f5debfff640 10 client.?.objecter ms_handle_connect 0x7f5e0413f710 2024-03-25T18:41:34.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.718+0000 7f5debfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:34.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.722+0000 7f5debfff640 10 client.?.objecter ms_handle_connect 0x7f5e041430c0 2024-03-25T18:41:34.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.722+0000 7f5debfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:34.725 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.726+0000 7f5debfff640 10 client.?.objecter ms_handle_connect 0x7f5e040a98f0 2024-03-25T18:41:34.726 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.726+0000 7f5debfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:34.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.729+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365395437 0 0) 0x7f5e041d7880 con 0x7f5e041430c0 2024-03-25T18:41:34.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.729+0000 7f5debfff640 10 monclient(hunting): _negotiate 2024-03-25T18:41:34.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.729+0000 7f5debfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:34.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.729+0000 7f5debfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:34.730 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.729+0000 7f5debfff640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:41:34.730 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.729+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5df0004eb0 con 0x7f5e041430c0 2024-03-25T18:41:34.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.731+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818726183 0 0) 0x7f5e040ab9e0 con 0x7f5e0413f710 2024-03-25T18:41:34.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.731+0000 7f5debfff640 10 monclient(hunting): _negotiate 2024-03-25T18:41:34.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.731+0000 7f5debfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:34.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.731+0000 7f5debfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:34.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.731+0000 7f5debfff640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:41:34.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.731+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e041d7880 con 0x7f5e0413f710 2024-03-25T18:41:34.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.737+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1588180889 0 0) 0x7f5e04138a50 con 0x7f5e040a98f0 2024-03-25T18:41:34.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.737+0000 7f5debfff640 10 monclient(hunting): _negotiate 2024-03-25T18:41:34.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.737+0000 7f5debfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:34.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.737+0000 7f5debfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:34.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.737+0000 7f5debfff640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:41:34.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.737+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e040ab9e0 con 0x7f5e040a98f0 2024-03-25T18:41:34.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.741+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1620617820 0 0) 0x7f5e041d7880 con 0x7f5e0413f710 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.741+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e04138a50 con 0x7f5e0413f710 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.741+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5df8001b40 con 0x7f5e0413f710 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.741+0000 7f5debfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.741+0000 7f5debfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.741+0000 7f5debfff640 10 monclient(hunting): dump: 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:34.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.741+0000 7f5debfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.751+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2493247479 0 0) 0x7f5e040ab9e0 con 0x7f5e040a98f0 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.751+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5e041d7880 con 0x7f5e040a98f0 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.751+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5df4001930 con 0x7f5e040a98f0 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.751+0000 7f5debfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.751+0000 7f5debfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.751+0000 7f5debfff640 10 monclient(hunting): dump: 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:34.751 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:34.752 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:34.752 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:34.752 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:34.752 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:34.752 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:34.752 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:34.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2912175072 0 0) 0x7f5e04138a50 con 0x7f5e0413f710 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 >> v1:172.21.15.32:6790/0 conn(0x7f5e041430c0 legacy=0x7f5e041e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 >> v1:172.21.15.32:6789/0 conn(0x7f5e040a98f0 legacy=0x7f5e04138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 10 monclient: _finish_hunting 0 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 1 monclient: found mon.b 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e041d84b0 con 0x7f5e0413f710 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 10 monclient: _finish_auth 0 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 10 monclient: _check_auth_tickets 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5debfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 5 monclient: authenticate success, global_id 4318 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 10 monclient: _renew_subs 2024-03-25T18:41:34.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:34.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e041d51f0 con 0x7f5e0413f710 2024-03-25T18:41:34.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 10 client.4318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:41:34.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 10 monclient: _renew_subs 2024-03-25T18:41:34.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:34.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e04138e60 con 0x7f5e0413f710 2024-03-25T18:41:34.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.757+0000 7f5e09863640 1 librados: init done 2024-03-25T18:41:34.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.758+0000 7f5e09863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:34.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.758+0000 7f5e09863640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:34.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.758+0000 7f5e09863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:34.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:34.758+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e041d76e0 con 0x7f5e0413f710 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.046+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5df80011f0 con 0x7f5e0413f710 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.046+0000 7f5debfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.046+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5df80038a0 con 0x7f5e0413f710 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.046+0000 7f5debfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.046+0000 7f5debfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.046+0000 7f5debfff640 10 monclient: dump: 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:35.047 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:35.048 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:35.048 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:35.048 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:35.048 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:35.048 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:35.048 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:35.048 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:35.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.046+0000 7f5e017fa640 4 set_mon_vals no callback set 2024-03-25T18:41:35.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.047+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5df8003ac0 con 0x7f5e0413f710 2024-03-25T18:41:35.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.048+0000 7f5e02ffd640 10 monclient: get_auth_request con 0x7f5df006df30 auth_method 0 2024-03-25T18:41:35.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.048+0000 7f5debfff640 10 client.4318.objecter ms_handle_connect 0x7f5df006df30 2024-03-25T18:41:35.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.052+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5df80546b0 con 0x7f5e0413f710 2024-03-25T18:41:35.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.052+0000 7f5debfff640 10 client.4318.objecter ms_dispatch 0x7f5e04000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:41:35.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.052+0000 7f5debfff640 3 client.4318.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:41:35.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.052+0000 7f5debfff640 3 client.4318.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:41:35.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.052+0000 7f5debfff640 20 client.4318.objecter dump_active .. 0 homeless 2024-03-25T18:41:35.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.052+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5df80501c0 con 0x7f5e0413f710 2024-03-25T18:41:35.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.053+0000 7f5debfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:35.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.053+0000 7f5debfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:41:35.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.312+0000 7f5e09863640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:35.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.312+0000 7f5e09863640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:35.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.312+0000 7f5e09863640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:35.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.312+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e041d7a10 con 0x7f5e0413f710 2024-03-25T18:41:35.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.349+0000 7f5debfff640 1 -- 172.21.15.32:0/479872967 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5df80520f0 con 0x7f5e0413f710 2024-03-25T18:41:35.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.349+0000 7f5debfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:35.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.349+0000 7f5debfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:41:35.350 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:41:35.350 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 10 librados: watch_flush enter 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 10 librados: watch_flush exit 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 20 client.4318.objecter shutdown clearing up homeless session... 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 10 client.4318.objecter successfully canceled tick 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 >> v1:172.21.15.136:6800/1141502808 conn(0x7f5df006df30 legacy=0x7f5df00703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 10 monclient: shutdown 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 >> v1:172.21.15.136:6789/0 conn(0x7f5e0413f710 legacy=0x7f5e041d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 shutdown_connections 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 >> 172.21.15.32:0/479872967 conn(0x7f5e040b7ea0 msgr2=0x7f5e04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 shutdown_connections 2024-03-25T18:41:35.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 1 -- 172.21.15.32:0/479872967 wait complete. 2024-03-25T18:41:35.354 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:35.352+0000 7f5e09863640 1 librados: shutdown 2024-03-25T18:41:35.365 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:41:42.172 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:41:42.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.281+0000 7f8123693640 10 monclient: get_monmap_and_config 2024-03-25T18:41:42.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.281+0000 7f8123693640 10 monclient: build_initial_monmap 2024-03-25T18:41:42.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.281+0000 7f8123693640 10 monclient: monmap: 2024-03-25T18:41:42.281 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:42.281 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:42.281 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:41:42.281799+0000 2024-03-25T18:41:42.282 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:41:42.281799+0000 2024-03-25T18:41:42.282 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:42.282 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:42.282 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:42.282 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:41:42.282 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:41:42.282 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:42.283 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.281+0000 7f8123693640 1 Processor -- start 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 1 -- start start 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 10 monclient: init 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 10 monclient(hunting): picked mon.noname-c con 0x7f811c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 10 monclient(hunting): picked mon.noname-a con 0x7f811c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 10 monclient(hunting): picked mon.noname-b con 0x7f811c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811c142520 con 0x7f811c13f710 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811c0b28a0 con 0x7f811c0a98f0 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811c0b2a30 con 0x7f811c1430c0 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.282+0000 7f8123693640 10 monclient(hunting): authenticate will time out at 6208.471272s 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.283+0000 7f8121c30640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f811c1430c0 0x7f811c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:42230/0 (socket says 172.21.15.32:42230) 2024-03-25T18:41:42.284 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.283+0000 7f8121c30640 1 -- 172.21.15.32:0/1064972157 learned_addr learned my addr 172.21.15.32:0/1064972157 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:42.285 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.283+0000 7f812142f640 1 --1- 172.21.15.32:0/1064972157 >> v1:172.21.15.136:6789/0 conn(0x7f811c0a98f0 0x7f811c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:44708/0 (socket says 172.21.15.32:44708) 2024-03-25T18:41:42.368 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.368+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1505780056 0 0) 0x7f811c142520 con 0x7f811c13f710 2024-03-25T18:41:42.368 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.368+0000 7f810ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:41:42.369 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.368+0000 7f810ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:42.369 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.368+0000 7f810ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:42.369 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.368+0000 7f810ffff640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:41:42.369 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.368+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8108003520 con 0x7f811c13f710 2024-03-25T18:41:42.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4163453400 0 0) 0x7f8108003520 con 0x7f811c13f710 2024-03-25T18:41:42.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 >> v1:172.21.15.32:6790/0 conn(0x7f811c1430c0 legacy=0x7f811c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:42.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 >> v1:172.21.15.136:6789/0 conn(0x7f811c0a98f0 legacy=0x7f811c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:42.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:41:42.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 1 monclient: found mon.noname-a 2024-03-25T18:41:42.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:42.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c0b44c0 con 0x7f811c13f710 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 10 monclient: _finish_auth 0 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8110001690 con 0x7f811c13f710 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f8123693640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f8123693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f810ffff640 10 monclient: dump: 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:42.384 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:42.385 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:42.385 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:42.385 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:42.385 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:42.385 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:42.385 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:42.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.383+0000 7f8123693640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.388+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81100017f0 con 0x7f811c13f710 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.388+0000 7f810ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.388+0000 7f8123693640 10 monclient: get_monmap_and_config success 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.388+0000 7f810ffff640 1 -- 172.21.15.32:0/1064972157 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8110001c00 con 0x7f811c13f710 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.388+0000 7f810ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.388+0000 7f8123693640 4 set_mon_vals no callback set 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.388+0000 7f810ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.388+0000 7f810ffff640 10 monclient: dump: 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:42.389 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:42.390 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:42.390 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:42.390 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:42.390 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:42.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.389+0000 7f8123693640 10 monclient: shutdown 2024-03-25T18:41:42.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.389+0000 7f8123693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:42.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.389+0000 7f8123693640 1 -- 172.21.15.32:0/1064972157 >> v1:172.21.15.32:6789/0 conn(0x7f811c13f710 legacy=0x7f811c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:42.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.389+0000 7f8123693640 1 -- 172.21.15.32:0/1064972157 shutdown_connections 2024-03-25T18:41:42.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.389+0000 7f8123693640 1 -- 172.21.15.32:0/1064972157 >> 172.21.15.32:0/1064972157 conn(0x7f811c0b7ea0 msgr2=0x7f811c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:42.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.389+0000 7f8123693640 1 -- 172.21.15.32:0/1064972157 shutdown_connections 2024-03-25T18:41:42.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.390+0000 7f8123693640 1 -- 172.21.15.32:0/1064972157 wait complete. 2024-03-25T18:41:42.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.390+0000 7f8123693640 10 monclient: build_initial_monmap 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.390+0000 7f8123693640 10 monclient: monmap: 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.390+0000 7f8123693640 1 librados: starting msgr at 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.390+0000 7f8123693640 1 librados: starting objecter 2024-03-25T18:41:42.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 1 Processor -- start 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 1 -- start start 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 1 librados: setting wanted keys 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 1 librados: calling monclient init 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 10 monclient: init 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 10 monclient(hunting): picked mon.noname-c con 0x7f811c0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 10 monclient(hunting): picked mon.noname-a con 0x7f811c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 10 monclient(hunting): picked mon.noname-b con 0x7f811c1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811c0abd70 con 0x7f811c13f710 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811c1d7710 con 0x7f811c1430c0 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f811c1d8340 con 0x7f811c0a98f0 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:42.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.391+0000 7f8123693640 10 monclient(hunting): authenticate will time out at 6208.580191s 2024-03-25T18:41:42.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.399+0000 7f8120c2e640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f811c13f710 0x7f811c1d7c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:51638/0 (socket says 172.21.15.32:51638) 2024-03-25T18:41:42.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.399+0000 7f8120c2e640 1 -- 172.21.15.32:0/3163247098 learned_addr learned my addr 172.21.15.32:0/3163247098 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:42.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.401+0000 7f810dffb640 10 client.?.objecter ms_handle_connect 0x7f811c13f710 2024-03-25T18:41:42.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.401+0000 7f810dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:42.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.406+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065025267 0 0) 0x7f811c0abd70 con 0x7f811c13f710 2024-03-25T18:41:42.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.406+0000 7f810dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:41:42.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.406+0000 7f810dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:42.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.406+0000 7f810dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:42.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.406+0000 7f810dffb640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:41:42.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.406+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8100004eb0 con 0x7f811c13f710 2024-03-25T18:41:42.419 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.419+0000 7f810dffb640 10 client.?.objecter ms_handle_connect 0x7f811c1430c0 2024-03-25T18:41:42.419 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.419+0000 7f810dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:42.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.427+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1250441067 0 0) 0x7f8100004eb0 con 0x7f811c13f710 2024-03-25T18:41:42.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.427+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f811c0abd70 con 0x7f811c13f710 2024-03-25T18:41:42.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.427+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8110001690 con 0x7f811c13f710 2024-03-25T18:41:42.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.427+0000 7f810dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:42.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.427+0000 7f810dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:41:42.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.427+0000 7f810dffb640 10 monclient(hunting): dump: 2024-03-25T18:41:42.429 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:42.429 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:42.429 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:42.429 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:42.430 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:42.430 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:42.430 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:42.430 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:42.430 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:42.430 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:42.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.427+0000 7f810dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:41:42.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.429+0000 7f810dffb640 10 client.?.objecter ms_handle_connect 0x7f811c0a98f0 2024-03-25T18:41:42.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.429+0000 7f810dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1448821471 0 0) 0x7f811c0abd70 con 0x7f811c13f710 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 >> v1:172.21.15.32:6790/0 conn(0x7f811c0a98f0 legacy=0x7f811c137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 >> v1:172.21.15.136:6789/0 conn(0x7f811c1430c0 legacy=0x7f811c1d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 1 monclient: found mon.a 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c1d84d0 con 0x7f811c13f710 2024-03-25T18:41:42.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 10 monclient: _finish_auth 0 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f810dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f8123693640 5 monclient: authenticate success, global_id 4335 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f8123693640 10 monclient: _renew_subs 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f8123693640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.443+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811c1d41e0 con 0x7f811c13f710 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.444+0000 7f8123693640 10 client.4335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.444+0000 7f8123693640 10 monclient: _renew_subs 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.444+0000 7f8123693640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.444+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f811c1d5150 con 0x7f811c13f710 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.444+0000 7f8123693640 1 librados: init done 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.445+0000 7f8123693640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.445+0000 7f8123693640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:42.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.445+0000 7f8123693640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:42.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.445+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811c0627b0 con 0x7f811c13f710 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.779+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81100038a0 con 0x7f811c13f710 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.779+0000 7f810dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.779+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8110003cb0 con 0x7f811c13f710 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.779+0000 7f810dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.779+0000 7f810dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.779+0000 7f810dffb640 10 monclient: dump: 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:42.780 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:42.781 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:42.781 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:42.781 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:42.781 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:42.781 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:42.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.779+0000 7f810f7fe640 4 set_mon_vals no callback set 2024-03-25T18:41:42.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.781+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f8110003ed0 con 0x7f811c13f710 2024-03-25T18:41:42.781 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.781+0000 7f812142f640 10 monclient: get_auth_request con 0x7f810006dc80 auth_method 0 2024-03-25T18:41:42.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.781+0000 7f810dffb640 10 client.4335.objecter ms_handle_connect 0x7f810006dc80 2024-03-25T18:41:42.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.787+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f8110054a10 con 0x7f811c13f710 2024-03-25T18:41:42.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.787+0000 7f810dffb640 10 client.4335.objecter ms_dispatch 0x7f811c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:41:42.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.787+0000 7f810dffb640 3 client.4335.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:41:42.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.787+0000 7f810dffb640 3 client.4335.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:41:42.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.787+0000 7f810dffb640 20 client.4335.objecter dump_active .. 0 homeless 2024-03-25T18:41:42.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.787+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8110050520 con 0x7f811c13f710 2024-03-25T18:41:42.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.787+0000 7f810dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:42.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:42.787+0000 7f810dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:41:43.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.038+0000 7f8123693640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:43.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.038+0000 7f8123693640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:43.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.038+0000 7f8123693640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:43.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.038+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f811c1d78a0 con 0x7f811c13f710 2024-03-25T18:41:43.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.065+0000 7f810dffb640 1 -- 172.21.15.32:0/3163247098 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f8110052450 con 0x7f811c13f710 2024-03-25T18:41:43.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.065+0000 7f810dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:43.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.065+0000 7f810dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:41:43.066 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:41:43.066 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:41:43.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 10 librados: watch_flush enter 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 10 librados: watch_flush exit 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 20 client.4335.objecter shutdown clearing up homeless session... 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 10 client.4335.objecter successfully canceled tick 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 >> v1:172.21.15.136:6800/1141502808 conn(0x7f810006dc80 legacy=0x7f8100070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 10 monclient: shutdown 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 >> v1:172.21.15.32:6789/0 conn(0x7f811c13f710 legacy=0x7f811c1d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 shutdown_connections 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 >> 172.21.15.32:0/3163247098 conn(0x7f811c0b7ea0 msgr2=0x7f811c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 shutdown_connections 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 1 -- 172.21.15.32:0/3163247098 wait complete. 2024-03-25T18:41:43.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:43.068+0000 7f8123693640 1 librados: shutdown 2024-03-25T18:41:43.081 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:41:47.562 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:59.890 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:47.570 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:59.898 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:47.599 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:02:59.927 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:47.621 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:59.949 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:47.627 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:59.955 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:47.636 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:59.953 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:47.644 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:59.961 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:47.669 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:02:59.997 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:47.673 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:02:59.990 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:41:49.887 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.005+0000 7f6321ed3640 10 monclient: get_monmap_and_config 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.005+0000 7f6321ed3640 10 monclient: build_initial_monmap 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.005+0000 7f6321ed3640 10 monclient: monmap: 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:41:50.006223+0000 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:41:50.006223+0000 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:50.006 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:50.007 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:41:50.007 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:41:50.007 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:50.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.006+0000 7f6321ed3640 1 Processor -- start 2024-03-25T18:41:50.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.006+0000 7f6321ed3640 1 -- start start 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.006+0000 7f6321ed3640 10 monclient: init 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.007+0000 7f6321ed3640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.007+0000 7f6321ed3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.007+0000 7f6321ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f631c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.007+0000 7f6321ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f631c13f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.007+0000 7f6321ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f631c1430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.008+0000 7f6321ed3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631c142520 con 0x7f631c1430c0 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.008+0000 7f6321ed3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631c0b28a0 con 0x7f631c0a98f0 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.008+0000 7f6321ed3640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631c0b2a30 con 0x7f631c13f710 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.008+0000 7f6321ed3640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:50.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.008+0000 7f6321ed3640 10 monclient(hunting): authenticate will time out at 6216.196690s 2024-03-25T18:41:50.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.011+0000 7f631bfff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f631c1430c0 0x7f631c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39734/0 (socket says 172.21.15.32:39734) 2024-03-25T18:41:50.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.011+0000 7f631bfff640 1 -- 172.21.15.32:0/1867310106 learned_addr learned my addr 172.21.15.32:0/1867310106 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:50.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.024+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028658804 0 0) 0x7f631c142520 con 0x7f631c1430c0 2024-03-25T18:41:50.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.024+0000 7f631a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:41:50.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.024+0000 7f631a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:50.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.024+0000 7f631a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:50.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.024+0000 7f631a7fc640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:41:50.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.024+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6304003520 con 0x7f631c1430c0 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1254988345 0 0) 0x7f6304003520 con 0x7f631c1430c0 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 >> v1:172.21.15.32:6790/0 conn(0x7f631c13f710 legacy=0x7f631c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 >> v1:172.21.15.136:6789/0 conn(0x7f631c0a98f0 legacy=0x7f631c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c0b44c0 con 0x7f631c1430c0 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:41:50.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6314001a10 con 0x7f631c1430c0 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f631a7fc640 10 monclient: dump: 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:50.035 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:50.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f6321ed3640 5 monclient: authenticate success, global_id 4338 2024-03-25T18:41:50.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.034+0000 7f6321ed3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:50.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.038+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6314001b70 con 0x7f631c1430c0 2024-03-25T18:41:50.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.038+0000 7f631a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:50.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.038+0000 7f6321ed3640 10 monclient: get_monmap_and_config success 2024-03-25T18:41:50.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.038+0000 7f6321ed3640 4 set_mon_vals no callback set 2024-03-25T18:41:50.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f631a7fc640 1 -- 172.21.15.32:0/1867310106 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6314001fc0 con 0x7f631c1430c0 2024-03-25T18:41:50.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f631a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:50.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f631a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:50.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f631a7fc640 10 monclient: dump: 2024-03-25T18:41:50.040 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:50.040 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f6321ed3640 10 monclient: shutdown 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f6321ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f6321ed3640 1 -- 172.21.15.32:0/1867310106 >> v1:172.21.15.32:6789/0 conn(0x7f631c1430c0 legacy=0x7f631c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f6321ed3640 1 -- 172.21.15.32:0/1867310106 shutdown_connections 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f6321ed3640 1 -- 172.21.15.32:0/1867310106 >> 172.21.15.32:0/1867310106 conn(0x7f631c0b7ea0 msgr2=0x7f631c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:50.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f6321ed3640 1 -- 172.21.15.32:0/1867310106 shutdown_connections 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.039+0000 7f6321ed3640 1 -- 172.21.15.32:0/1867310106 wait complete. 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.040+0000 7f6321ed3640 10 monclient: build_initial_monmap 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.040+0000 7f6321ed3640 10 monclient: monmap: 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:41:50.042 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.041+0000 7f6321ed3640 1 librados: starting msgr at 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.041+0000 7f6321ed3640 1 librados: starting objecter 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.041+0000 7f6321ed3640 1 Processor -- start 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.041+0000 7f6321ed3640 1 -- start start 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.041+0000 7f6321ed3640 1 librados: setting wanted keys 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.041+0000 7f6321ed3640 1 librados: calling monclient init 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.041+0000 7f6321ed3640 10 monclient: init 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:41:50.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 10 monclient(hunting): picked mon.noname-a con 0x7f631c0a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:50.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 10 monclient(hunting): picked mon.noname-c con 0x7f631c13f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:50.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 10 monclient(hunting): picked mon.noname-b con 0x7f631c1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:50.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631c1dacd0 con 0x7f631c0a98f0 2024-03-25T18:41:50.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631c1d8230 con 0x7f631c1430c0 2024-03-25T18:41:50.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631c1da8f0 con 0x7f631c13f710 2024-03-25T18:41:50.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:50.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.042+0000 7f6321ed3640 10 monclient(hunting): authenticate will time out at 6216.231014s 2024-03-25T18:41:50.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.046+0000 7f631b7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f631c0a98f0 0x7f631c1d73f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39750/0 (socket says 172.21.15.32:39750) 2024-03-25T18:41:50.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.046+0000 7f631b7fe640 1 -- 172.21.15.32:0/2084161035 learned_addr learned my addr 172.21.15.32:0/2084161035 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:50.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.047+0000 7f6303fff640 10 client.?.objecter ms_handle_connect 0x7f631c0a98f0 2024-03-25T18:41:50.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.047+0000 7f6303fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:50.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.053+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1097387919 0 0) 0x7f631c1dacd0 con 0x7f631c0a98f0 2024-03-25T18:41:50.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.053+0000 7f6303fff640 10 monclient(hunting): _negotiate 2024-03-25T18:41:50.053 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.053+0000 7f6303fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:50.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.053+0000 7f6303fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:50.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.053+0000 7f6303fff640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:41:50.054 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.053+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6308004eb0 con 0x7f631c0a98f0 2024-03-25T18:41:50.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.058+0000 7f6303fff640 10 client.?.objecter ms_handle_connect 0x7f631c1430c0 2024-03-25T18:41:50.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.058+0000 7f6303fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:50.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.059+0000 7f6303fff640 10 client.?.objecter ms_handle_connect 0x7f631c13f710 2024-03-25T18:41:50.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.059+0000 7f6303fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.063+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4216573848 0 0) 0x7f6308004eb0 con 0x7f631c0a98f0 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.063+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f631c1dacd0 con 0x7f631c0a98f0 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.063+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f630c001a30 con 0x7f631c0a98f0 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.063+0000 7f6303fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.064+0000 7f6303fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.064+0000 7f6303fff640 10 monclient(hunting): dump: 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:50.064 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:50.065 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:50.065 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:50.065 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:50.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.064+0000 7f6303fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:41:50.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.076+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3247142035 0 0) 0x7f631c1dacd0 con 0x7f631c0a98f0 2024-03-25T18:41:50.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.076+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 >> v1:172.21.15.32:6790/0 conn(0x7f631c13f710 legacy=0x7f631c1db1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:50.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.076+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 >> v1:172.21.15.136:6789/0 conn(0x7f631c1430c0 legacy=0x7f631c138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:50.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6303fff640 10 monclient: _finish_hunting 0 2024-03-25T18:41:50.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6303fff640 1 monclient: found mon.a 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6303fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6303fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631c1daa80 con 0x7f631c0a98f0 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6303fff640 10 monclient: _finish_auth 0 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6303fff640 10 monclient: _check_auth_tickets 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6303fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 5 monclient: authenticate success, global_id 4341 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 10 monclient: _renew_subs 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631c137930 con 0x7f631c0a98f0 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 10 monclient: _renew_subs 2024-03-25T18:41:50.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:50.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f631c137f60 con 0x7f631c0a98f0 2024-03-25T18:41:50.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.077+0000 7f6321ed3640 1 librados: init done 2024-03-25T18:41:50.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.078+0000 7f6321ed3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:50.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.078+0000 7f6321ed3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:50.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.078+0000 7f6321ed3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:50.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.078+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631c0627b0 con 0x7f631c0a98f0 2024-03-25T18:41:50.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.393+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f630c001cb0 con 0x7f631c0a98f0 2024-03-25T18:41:50.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.393+0000 7f6303fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:50.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.393+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f630c003a60 con 0x7f631c0a98f0 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.393+0000 7f6303fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.393+0000 7f6303fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.393+0000 7f6303fff640 10 monclient: dump: 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:50.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.393+0000 7f6319ffb640 4 set_mon_vals no callback set 2024-03-25T18:41:50.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.394+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f630c003c40 con 0x7f631c0a98f0 2024-03-25T18:41:50.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.394+0000 7f631affd640 10 monclient: get_auth_request con 0x7f630806dc20 auth_method 0 2024-03-25T18:41:50.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.394+0000 7f6303fff640 10 client.4341.objecter ms_handle_connect 0x7f630806dc20 2024-03-25T18:41:50.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.399+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f630c054880 con 0x7f631c0a98f0 2024-03-25T18:41:50.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.400+0000 7f6303fff640 10 client.4341.objecter ms_dispatch 0x7f631c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:41:50.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.400+0000 7f6303fff640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:41:50.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.400+0000 7f6303fff640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:41:50.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.400+0000 7f6303fff640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-25T18:41:50.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.400+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f630c050390 con 0x7f631c0a98f0 2024-03-25T18:41:50.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.400+0000 7f6303fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:50.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.400+0000 7f6303fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:41:50.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.645+0000 7f6321ed3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:50.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.645+0000 7f6321ed3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:50.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.645+0000 7f6321ed3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:41:50.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.645+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f631c069f40 con 0x7f631c0a98f0 2024-03-25T18:41:50.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.738+0000 7f6303fff640 1 -- 172.21.15.32:0/2084161035 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f630c0522c0 con 0x7f631c0a98f0 2024-03-25T18:41:50.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.738+0000 7f6303fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:50.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.738+0000 7f6303fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:41:50.738 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:41:50.739 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 10 librados: watch_flush enter 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 10 librados: watch_flush exit 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 10 client.4341.objecter successfully canceled tick 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 >> v1:172.21.15.136:6800/1141502808 conn(0x7f630806dc20 legacy=0x7f63080700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 10 monclient: shutdown 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 >> v1:172.21.15.32:6789/0 conn(0x7f631c0a98f0 legacy=0x7f631c1d73f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 shutdown_connections 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 >> 172.21.15.32:0/2084161035 conn(0x7f631c0b7ea0 msgr2=0x7f631c13c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:50.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 shutdown_connections 2024-03-25T18:41:50.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 1 -- 172.21.15.32:0/2084161035 wait complete. 2024-03-25T18:41:50.742 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:50.740+0000 7f6321ed3640 1 librados: shutdown 2024-03-25T18:41:50.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:41:57.558 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:41:57.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.675+0000 7fc561777640 10 monclient: get_monmap_and_config 2024-03-25T18:41:57.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.675+0000 7fc561777640 10 monclient: build_initial_monmap 2024-03-25T18:41:57.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.675+0000 7fc561777640 10 monclient: monmap: 2024-03-25T18:41:57.677 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:57.677 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:57.677 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:41:57.677366+0000 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:41:57.677366+0000 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.676+0000 7fc561777640 1 Processor -- start 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.676+0000 7fc561777640 1 -- start start 2024-03-25T18:41:57.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.676+0000 7fc561777640 10 monclient: init 2024-03-25T18:41:57.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:57.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:41:57.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 10 monclient(hunting): picked mon.noname-a con 0x7fc55c0a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:57.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 10 monclient(hunting): picked mon.noname-c con 0x7fc55c13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:57.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 10 monclient(hunting): picked mon.noname-b con 0x7fc55c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:57.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c142520 con 0x7fc55c0a98f0 2024-03-25T18:41:57.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0b28a0 con 0x7fc55c13f710 2024-03-25T18:41:57.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0b2a30 con 0x7fc55c1430c0 2024-03-25T18:41:57.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:57.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.677+0000 7fc561777640 10 monclient(hunting): authenticate will time out at 6223.866992s 2024-03-25T18:41:57.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.684+0000 7fc55b7fe640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fc55c1430c0 0x7fc55c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:51196/0 (socket says 172.21.15.32:51196) 2024-03-25T18:41:57.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.684+0000 7fc55b7fe640 1 -- 172.21.15.32:0/3050140977 learned_addr learned my addr 172.21.15.32:0/3050140977 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:57.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.696+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3422957389 0 0) 0x7fc55c0b2a30 con 0x7fc55c1430c0 2024-03-25T18:41:57.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.696+0000 7fc559ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:41:57.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.696+0000 7fc559ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:57.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.696+0000 7fc559ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:57.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.696+0000 7fc559ffb640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:41:57.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.696+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc544003520 con 0x7fc55c1430c0 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2521792366 0 0) 0x7fc544003520 con 0x7fc55c1430c0 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 >> v1:172.21.15.136:6789/0 conn(0x7fc55c13f710 legacy=0x7fc55c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 >> v1:172.21.15.32:6789/0 conn(0x7fc55c0a98f0 legacy=0x7fc55c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 1 monclient: found mon.noname-b 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0b44c0 con 0x7fc55c1430c0 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 10 monclient: _finish_auth 0 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:41:57.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc554001a10 con 0x7fc55c1430c0 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc559ffb640 10 monclient: dump: 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:57.708 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:57.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc561777640 5 monclient: authenticate success, global_id 4334 2024-03-25T18:41:57.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.705+0000 7fc561777640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:41:57.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.713+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc554001b70 con 0x7fc55c1430c0 2024-03-25T18:41:57.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.713+0000 7fc559ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.713+0000 7fc559ffb640 1 -- 172.21.15.32:0/3050140977 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc554001f80 con 0x7fc55c1430c0 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.713+0000 7fc559ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.713+0000 7fc559ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.713+0000 7fc559ffb640 10 monclient: dump: 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:57.715 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:57.716 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:57.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.713+0000 7fc561777640 10 monclient: get_monmap_and_config success 2024-03-25T18:41:57.716 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.713+0000 7fc561777640 4 set_mon_vals no callback set 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.714+0000 7fc561777640 10 monclient: shutdown 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.714+0000 7fc561777640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.714+0000 7fc561777640 1 -- 172.21.15.32:0/3050140977 >> v1:172.21.15.32:6790/0 conn(0x7fc55c1430c0 legacy=0x7fc55c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.714+0000 7fc561777640 1 -- 172.21.15.32:0/3050140977 shutdown_connections 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.714+0000 7fc561777640 1 -- 172.21.15.32:0/3050140977 >> 172.21.15.32:0/3050140977 conn(0x7fc55c0b7ea0 msgr2=0x7fc55c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.714+0000 7fc561777640 1 -- 172.21.15.32:0/3050140977 shutdown_connections 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.714+0000 7fc561777640 1 -- 172.21.15.32:0/3050140977 wait complete. 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 10 monclient: build_initial_monmap 2024-03-25T18:41:57.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 10 monclient: monmap: 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 1 librados: starting msgr at 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 1 librados: starting objecter 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 1 Processor -- start 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 1 -- start start 2024-03-25T18:41:57.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 1 librados: setting wanted keys 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 1 librados: calling monclient init 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.715+0000 7fc561777640 10 monclient: init 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 10 monclient: _reopen_session rank -1 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 10 monclient(hunting): picked mon.noname-b con 0x7fc55c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 10 monclient(hunting): picked mon.noname-a con 0x7fc55c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 10 monclient(hunting): picked mon.noname-c con 0x7fc55c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c1369d0 con 0x7fc55c13f710 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c133a60 con 0x7fc55c0a98f0 2024-03-25T18:41:57.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc55c0ab9e0 con 0x7fc55c1430c0 2024-03-25T18:41:57.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 10 monclient(hunting): _renew_subs 2024-03-25T18:41:57.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.716+0000 7fc561777640 10 monclient(hunting): authenticate will time out at 6223.906352s 2024-03-25T18:41:57.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.722+0000 7fc55a7fc640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fc55c13f710 0x7fc55c1362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:49138/0 (socket says 172.21.15.32:49138) 2024-03-25T18:41:57.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.722+0000 7fc55a7fc640 1 -- 172.21.15.32:0/333177346 learned_addr learned my addr 172.21.15.32:0/333177346 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:41:57.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.724+0000 7fc5437fe640 10 client.?.objecter ms_handle_connect 0x7fc55c13f710 2024-03-25T18:41:57.725 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.724+0000 7fc5437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:57.730 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.729+0000 7fc5437fe640 10 client.?.objecter ms_handle_connect 0x7fc55c0a98f0 2024-03-25T18:41:57.730 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.729+0000 7fc5437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:57.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.734+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528623477 0 0) 0x7fc55c133a60 con 0x7fc55c0a98f0 2024-03-25T18:41:57.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.734+0000 7fc5437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:41:57.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.734+0000 7fc5437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:57.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.734+0000 7fc5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:57.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.734+0000 7fc5437fe640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:41:57.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.734+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc548004eb0 con 0x7fc55c0a98f0 2024-03-25T18:41:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.740+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138634370 0 0) 0x7fc55c1369d0 con 0x7fc55c13f710 2024-03-25T18:41:57.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.740+0000 7fc5437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:41:57.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.740+0000 7fc5437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:57.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.740+0000 7fc5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:57.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.740+0000 7fc5437fe640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:41:57.741 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.740+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc55c133a60 con 0x7fc55c13f710 2024-03-25T18:41:57.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3888346598 0 0) 0x7fc548004eb0 con 0x7fc55c0a98f0 2024-03-25T18:41:57.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc55c1369d0 con 0x7fc55c0a98f0 2024-03-25T18:41:57.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc54c001930 con 0x7fc55c0a98f0 2024-03-25T18:41:57.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:57.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:41:57.751 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 10 monclient(hunting): dump: 2024-03-25T18:41:57.751 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:57.751 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 10 client.?.objecter ms_handle_connect 0x7fc55c1430c0 2024-03-25T18:41:57.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.750+0000 7fc5437fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:41:57.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.755+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 885962864 0 0) 0x7fc55c0ab9e0 con 0x7fc55c1430c0 2024-03-25T18:41:57.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.755+0000 7fc5437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:41:57.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.755+0000 7fc5437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:41:57.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.755+0000 7fc5437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:41:57.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.755+0000 7fc5437fe640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:41:57.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.755+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc548004eb0 con 0x7fc55c1430c0 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.756+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1551476399 0 0) 0x7fc55c133a60 con 0x7fc55c13f710 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.756+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc55c0ab9e0 con 0x7fc55c13f710 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.756+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc550001990 con 0x7fc55c13f710 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.756+0000 7fc5437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.756+0000 7fc5437fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.756+0000 7fc5437fe640 10 monclient(hunting): dump: 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:57.757 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:57.758 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:57.758 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:57.758 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:57.758 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:57.758 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 699563007 0 0) 0x7fc55c1369d0 con 0x7fc55c0a98f0 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 >> v1:172.21.15.32:6790/0 conn(0x7fc55c1430c0 legacy=0x7fc55c133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 >> v1:172.21.15.32:6789/0 conn(0x7fc55c13f710 legacy=0x7fc55c1362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 10 monclient: _finish_hunting 0 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 1 monclient: found mon.b 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc55c0abcc0 con 0x7fc55c0a98f0 2024-03-25T18:41:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 10 monclient: _finish_auth 0 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 10 monclient: _check_auth_tickets 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc5437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc561777640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc561777640 10 monclient: _renew_subs 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc561777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc55c0abfa0 con 0x7fc55c0a98f0 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc561777640 10 client.4330.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc561777640 10 monclient: _renew_subs 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc561777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.759+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc55c137e40 con 0x7fc55c0a98f0 2024-03-25T18:41:57.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.760+0000 7fc561777640 1 librados: init done 2024-03-25T18:41:57.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.761+0000 7fc561777640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:57.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.761+0000 7fc561777640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:57.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.761+0000 7fc561777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:57.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:57.761+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc55c139060 con 0x7fc55c0a98f0 2024-03-25T18:41:58.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.024+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc54c0035f0 con 0x7fc55c0a98f0 2024-03-25T18:41:58.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.024+0000 7fc5437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:41:58.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.024+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc54c003980 con 0x7fc55c0a98f0 2024-03-25T18:41:58.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.024+0000 7fc5437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:41:58.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.024+0000 7fc5437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:41:58.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.024+0000 7fc5437fe640 10 monclient: dump: 2024-03-25T18:41:58.026 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:41:58.026 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:41:58.026 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:58.026 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:41:58.026 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:41:58.027 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:41:58.027 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:41:58.027 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:41:58.027 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:41:58.027 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:41:58.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.024+0000 7fc5597fa640 4 set_mon_vals no callback set 2024-03-25T18:41:58.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.024+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fc54c003ba0 con 0x7fc55c0a98f0 2024-03-25T18:41:58.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.025+0000 7fc55a7fc640 10 monclient: get_auth_request con 0x7fc54806df70 auth_method 0 2024-03-25T18:41:58.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.025+0000 7fc5437fe640 10 client.4330.objecter ms_handle_connect 0x7fc54806df70 2024-03-25T18:41:58.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.027+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fc54c0535b0 con 0x7fc55c0a98f0 2024-03-25T18:41:58.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.027+0000 7fc5437fe640 10 client.4330.objecter ms_dispatch 0x7fc55c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:41:58.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.027+0000 7fc5437fe640 3 client.4330.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:41:58.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.027+0000 7fc5437fe640 3 client.4330.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:41:58.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.027+0000 7fc5437fe640 20 client.4330.objecter dump_active .. 0 homeless 2024-03-25T18:41:58.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.029+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc54c04f0c0 con 0x7fc55c0a98f0 2024-03-25T18:41:58.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.029+0000 7fc5437fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:41:58.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.029+0000 7fc5437fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:41:58.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.271+0000 7fc561777640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:58.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.271+0000 7fc561777640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:58.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.271+0000 7fc561777640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:41:58.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.271+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc55c0627b0 con 0x7fc55c0a98f0 2024-03-25T18:41:58.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.355+0000 7fc5437fe640 1 -- 172.21.15.32:0/333177346 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fc54c050ff0 con 0x7fc55c0a98f0 2024-03-25T18:41:58.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.355+0000 7fc5437fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:41:58.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.355+0000 7fc5437fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:41:58.356 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:41:58.357 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 10 librados: watch_flush enter 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 10 librados: watch_flush exit 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 20 client.4330.objecter shutdown clearing up homeless session... 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 10 client.4330.objecter successfully canceled tick 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 >> v1:172.21.15.136:6800/1141502808 conn(0x7fc54806df70 legacy=0x7fc548070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 10 monclient: shutdown 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 >> v1:172.21.15.136:6789/0 conn(0x7fc55c0a98f0 legacy=0x7fc55c132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 shutdown_connections 2024-03-25T18:41:58.359 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 >> 172.21.15.32:0/333177346 conn(0x7fc55c0b7ea0 msgr2=0x7fc55c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:41:58.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 shutdown_connections 2024-03-25T18:41:58.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 1 -- 172.21.15.32:0/333177346 wait complete. 2024-03-25T18:41:58.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:41:58.357+0000 7fc561777640 1 librados: shutdown 2024-03-25T18:41:58.370 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:42:05.178 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:42:05.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.292+0000 7f09ffd54640 10 monclient: get_monmap_and_config 2024-03-25T18:42:05.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.293+0000 7f09ffd54640 10 monclient: build_initial_monmap 2024-03-25T18:42:05.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.293+0000 7f09ffd54640 10 monclient: monmap: 2024-03-25T18:42:05.294 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:05.294 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:05.294 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:42:05.294484+0000 2024-03-25T18:42:05.294 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:42:05.294484+0000 2024-03-25T18:42:05.295 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:05.295 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:05.295 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:05.295 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:42:05.295 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:42:05.295 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:05.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.293+0000 7f09ffd54640 1 Processor -- start 2024-03-25T18:42:05.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.293+0000 7f09ffd54640 1 -- start start 2024-03-25T18:42:05.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.293+0000 7f09ffd54640 10 monclient: init 2024-03-25T18:42:05.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 10 monclient(hunting): picked mon.noname-b con 0x7f09f80a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 10 monclient(hunting): picked mon.noname-a con 0x7f09f813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 10 monclient(hunting): picked mon.noname-c con 0x7f09f81430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f8142520 con 0x7f09f813f710 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f80b28a0 con 0x7f09f81430c0 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f80b2a30 con 0x7f09f80a98f0 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.294+0000 7f09ffd54640 10 monclient(hunting): authenticate will time out at 6231.484147s 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.295+0000 7f09fdaf0640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f09f80a98f0 0x7f09f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:56050/0 (socket says 172.21.15.32:56050) 2024-03-25T18:42:05.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.295+0000 7f09fdaf0640 1 -- 172.21.15.32:0/1990323154 learned_addr learned my addr 172.21.15.32:0/1990323154 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:05.355 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.354+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862224089 0 0) 0x7f09f80b28a0 con 0x7f09f81430c0 2024-03-25T18:42:05.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.354+0000 7f09fcaee640 10 monclient(hunting): _negotiate 2024-03-25T18:42:05.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.354+0000 7f09fcaee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:05.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.354+0000 7f09fcaee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:05.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.354+0000 7f09fcaee640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:42:05.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.354+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4003520 con 0x7f09f81430c0 2024-03-25T18:42:05.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.371+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 726687849 0 0) 0x7f09f80b2a30 con 0x7f09f80a98f0 2024-03-25T18:42:05.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.371+0000 7f09fcaee640 10 monclient(hunting): _negotiate 2024-03-25T18:42:05.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.371+0000 7f09fcaee640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:05.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.371+0000 7f09fcaee640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:05.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.371+0000 7f09fcaee640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:42:05.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.371+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f80b28a0 con 0x7f09f80a98f0 2024-03-25T18:42:05.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2393593627 0 0) 0x7f09e4003520 con 0x7f09f81430c0 2024-03-25T18:42:05.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 >> v1:172.21.15.32:6790/0 conn(0x7f09f80a98f0 legacy=0x7f09f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:05.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 >> v1:172.21.15.32:6789/0 conn(0x7f09f813f710 legacy=0x7f09f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:05.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 10 monclient: _finish_hunting 0 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 1 monclient: found mon.noname-c 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f80b44c0 con 0x7f09f81430c0 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 10 monclient: _finish_auth 0 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 10 monclient: _check_auth_tickets 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f09f0001a10 con 0x7f09f81430c0 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:05.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.374+0000 7f09fcaee640 10 monclient: dump: 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:05.378 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:05.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.375+0000 7f09ffd54640 5 monclient: authenticate success, global_id 4333 2024-03-25T18:42:05.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.375+0000 7f09ffd54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:05.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f0001b70 con 0x7f09f81430c0 2024-03-25T18:42:05.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09fcaee640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:05.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09fcaee640 1 -- 172.21.15.32:0/1990323154 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f09f0001f80 con 0x7f09f81430c0 2024-03-25T18:42:05.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09ffd54640 10 monclient: get_monmap_and_config success 2024-03-25T18:42:05.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09ffd54640 4 set_mon_vals no callback set 2024-03-25T18:42:05.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09fcaee640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:05.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09fcaee640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:05.385 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09fcaee640 10 monclient: dump: 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:05.386 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09ffd54640 10 monclient: shutdown 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09ffd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.384+0000 7f09ffd54640 1 -- 172.21.15.32:0/1990323154 >> v1:172.21.15.136:6789/0 conn(0x7f09f81430c0 legacy=0x7f09f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.385+0000 7f09ffd54640 1 -- 172.21.15.32:0/1990323154 shutdown_connections 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.385+0000 7f09ffd54640 1 -- 172.21.15.32:0/1990323154 >> 172.21.15.32:0/1990323154 conn(0x7f09f80b7ea0 msgr2=0x7f09f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.385+0000 7f09ffd54640 1 -- 172.21.15.32:0/1990323154 shutdown_connections 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.385+0000 7f09ffd54640 1 -- 172.21.15.32:0/1990323154 wait complete. 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.385+0000 7f09ffd54640 10 monclient: build_initial_monmap 2024-03-25T18:42:05.389 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.385+0000 7f09ffd54640 10 monclient: monmap: 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.386+0000 7f09ffd54640 1 librados: starting msgr at 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.386+0000 7f09ffd54640 1 librados: starting objecter 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.386+0000 7f09ffd54640 1 Processor -- start 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.386+0000 7f09ffd54640 1 -- start start 2024-03-25T18:42:05.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.386+0000 7f09ffd54640 1 librados: setting wanted keys 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.386+0000 7f09ffd54640 1 librados: calling monclient init 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.386+0000 7f09ffd54640 10 monclient: init 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 10 monclient(hunting): picked mon.noname-b con 0x7f09f80a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 10 monclient(hunting): picked mon.noname-a con 0x7f09f813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 10 monclient(hunting): picked mon.noname-c con 0x7f09f81430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f8137c60 con 0x7f09f813f710 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f8139160 con 0x7f09f80a98f0 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f81380f0 con 0x7f09f81430c0 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:05.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.387+0000 7f09ffd54640 10 monclient(hunting): authenticate will time out at 6231.577445s 2024-03-25T18:42:05.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.390+0000 7f09fd2ef640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f09f813f710 0x7f09f8133070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53992/0 (socket says 172.21.15.32:53992) 2024-03-25T18:42:05.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.390+0000 7f09fd2ef640 1 -- 172.21.15.32:0/3830227730 learned_addr learned my addr 172.21.15.32:0/3830227730 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:05.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.391+0000 7f09ee7fc640 10 client.?.objecter ms_handle_connect 0x7f09f813f710 2024-03-25T18:42:05.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.391+0000 7f09ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:05.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.394+0000 7f09ee7fc640 10 client.?.objecter ms_handle_connect 0x7f09f81430c0 2024-03-25T18:42:05.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.394+0000 7f09ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:05.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.396+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1042706653 0 0) 0x7f09f8137c60 con 0x7f09f813f710 2024-03-25T18:42:05.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.396+0000 7f09ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:42:05.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.396+0000 7f09ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:05.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.396+0000 7f09ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:05.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.396+0000 7f09ee7fc640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:42:05.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.396+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09dc004eb0 con 0x7f09f813f710 2024-03-25T18:42:05.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.396+0000 7f09ee7fc640 10 client.?.objecter ms_handle_connect 0x7f09f80a98f0 2024-03-25T18:42:05.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.396+0000 7f09ee7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:05.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.404+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120132964 0 0) 0x7f09f81380f0 con 0x7f09f81430c0 2024-03-25T18:42:05.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.404+0000 7f09ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:42:05.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.404+0000 7f09ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:05.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.404+0000 7f09ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:05.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.404+0000 7f09ee7fc640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:42:05.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.404+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f8137c60 con 0x7f09f81430c0 2024-03-25T18:42:05.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.405+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3944244654 0 0) 0x7f09dc004eb0 con 0x7f09f813f710 2024-03-25T18:42:05.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.405+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09f81380f0 con 0x7f09f813f710 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.405+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f09e8001a30 con 0x7f09f813f710 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.405+0000 7f09ee7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.405+0000 7f09ee7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.405+0000 7f09ee7fc640 10 monclient(hunting): dump: 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:05.407 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:05.408 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:05.408 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:05.408 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.405+0000 7f09ee7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:42:05.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.408+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3018356450 0 0) 0x7f09f8139160 con 0x7f09f80a98f0 2024-03-25T18:42:05.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.408+0000 7f09ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:42:05.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.408+0000 7f09ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:05.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.408+0000 7f09ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:05.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.408+0000 7f09ee7fc640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:42:05.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.408+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09dc004eb0 con 0x7f09f80a98f0 2024-03-25T18:42:05.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2963568288 0 0) 0x7f09f81380f0 con 0x7f09f813f710 2024-03-25T18:42:05.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 >> v1:172.21.15.32:6790/0 conn(0x7f09f81430c0 legacy=0x7f09f81e2eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:05.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 >> v1:172.21.15.136:6789/0 conn(0x7f09f80a98f0 legacy=0x7f09f81dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:05.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 1 monclient: found mon.a 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f81397f0 con 0x7f09f813f710 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.411+0000 7f09ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 5 monclient: authenticate success, global_id 4353 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 10 monclient: _renew_subs 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f81db280 con 0x7f09f813f710 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 10 client.4353.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 10 monclient: _renew_subs 2024-03-25T18:42:05.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:05.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f8133960 con 0x7f09f813f710 2024-03-25T18:42:05.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.412+0000 7f09ffd54640 1 librados: init done 2024-03-25T18:42:05.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.413+0000 7f09ffd54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:05.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.413+0000 7f09ffd54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:05.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.413+0000 7f09ffd54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:05.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.413+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f8139a40 con 0x7f09f813f710 2024-03-25T18:42:05.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.735+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09e8001c70 con 0x7f09f813f710 2024-03-25T18:42:05.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.735+0000 7f09ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:05.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.735+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f09e8003a60 con 0x7f09f813f710 2024-03-25T18:42:05.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.735+0000 7f09ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:05.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.735+0000 7f09ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:42:05.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.735+0000 7f09ee7fc640 10 monclient: dump: 2024-03-25T18:42:05.737 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:05.737 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.735+0000 7f09effff640 4 set_mon_vals no callback set 2024-03-25T18:42:05.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.736+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f09e8003c40 con 0x7f09f813f710 2024-03-25T18:42:05.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.737+0000 7f09fdaf0640 10 monclient: get_auth_request con 0x7f09dc06de00 auth_method 0 2024-03-25T18:42:05.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.737+0000 7f09ee7fc640 10 client.4353.objecter ms_handle_connect 0x7f09dc06de00 2024-03-25T18:42:05.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.744+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f09e80548b0 con 0x7f09f813f710 2024-03-25T18:42:05.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.744+0000 7f09ee7fc640 10 client.4353.objecter ms_dispatch 0x7f09f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:42:05.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.744+0000 7f09ee7fc640 3 client.4353.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:42:05.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.744+0000 7f09ee7fc640 3 client.4353.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:42:05.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.744+0000 7f09ee7fc640 20 client.4353.objecter dump_active .. 0 homeless 2024-03-25T18:42:05.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.744+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09e80503c0 con 0x7f09f813f710 2024-03-25T18:42:05.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.744+0000 7f09ee7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:05.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.744+0000 7f09ee7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:42:05.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.992+0000 7f09ffd54640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:05.994 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.992+0000 7f09ffd54640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:05.994 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.992+0000 7f09ffd54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:05.994 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:05.992+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09f80627b0 con 0x7f09f813f710 2024-03-25T18:42:06.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.011+0000 7f09ee7fc640 1 -- 172.21.15.32:0/3830227730 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f09e80522f0 con 0x7f09f813f710 2024-03-25T18:42:06.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.011+0000 7f09ee7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:06.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.011+0000 7f09ee7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:42:06.013 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:42:06.013 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 10 librados: watch_flush enter 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 10 librados: watch_flush exit 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 20 client.4353.objecter shutdown clearing up homeless session... 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 10 client.4353.objecter successfully canceled tick 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 >> v1:172.21.15.136:6800/1141502808 conn(0x7f09dc06de00 legacy=0x7f09dc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 10 monclient: shutdown 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 >> v1:172.21.15.32:6789/0 conn(0x7f09f813f710 legacy=0x7f09f8133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 shutdown_connections 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 >> 172.21.15.32:0/3830227730 conn(0x7f09f80b7ea0 msgr2=0x7f09f813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:06.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.014+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 shutdown_connections 2024-03-25T18:42:06.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.015+0000 7f09ffd54640 1 -- 172.21.15.32:0/3830227730 wait complete. 2024-03-25T18:42:06.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:06.015+0000 7f09ffd54640 1 librados: shutdown 2024-03-25T18:42:06.028 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:42:07.437 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:19.753 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:07.437 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:19.765 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:07.439 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:19.756 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:07.459 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:19.787 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:07.484 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:19.801 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:07.492 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:19.820 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:07.500 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:03:19.828 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:07.509 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:03:19.837 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:07.534 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:03:19.862 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:12.835 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.949+0000 7ff209afb640 10 monclient: get_monmap_and_config 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.949+0000 7ff209afb640 10 monclient: build_initial_monmap 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.949+0000 7ff209afb640 10 monclient: monmap: 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:42:12.951432+0000 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:42:12.951432+0000 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:12.951 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:42:12.952 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:42:12.952 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.950+0000 7ff209afb640 1 Processor -- start 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 1 -- start start 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 10 monclient: init 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 10 monclient(hunting): picked mon.noname-b con 0x7ff2040b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 10 monclient(hunting): picked mon.noname-c con 0x7ff2040b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 10 monclient(hunting): picked mon.noname-a con 0x7ff20414be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2040af940 con 0x7ff20414be90 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2040b0a70 con 0x7ff2040b1730 2024-03-25T18:42:12.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2040b0d50 con 0x7ff2040b4d60 2024-03-25T18:42:12.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:12.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.951+0000 7ff209afb640 10 monclient(hunting): authenticate will time out at 6239.141471s 2024-03-25T18:42:12.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.959+0000 7ff2037fe640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7ff2040b4d60 0x7ff20413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:51404/0 (socket says 172.21.15.32:51404) 2024-03-25T18:42:12.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.959+0000 7ff2037fe640 1 -- 172.21.15.32:0/4230651775 learned_addr learned my addr 172.21.15.32:0/4230651775 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:12.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.972+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 261614025 0 0) 0x7ff2040b0d50 con 0x7ff2040b4d60 2024-03-25T18:42:12.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.972+0000 7ff202ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:12.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.972+0000 7ff202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:12.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.972+0000 7ff202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:12.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.972+0000 7ff202ffd640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:42:12.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.972+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1f0003520 con 0x7ff2040b4d60 2024-03-25T18:42:12.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.981+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4265047636 0 0) 0x7ff2040af940 con 0x7ff20414be90 2024-03-25T18:42:12.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.981+0000 7ff202ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:12.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.981+0000 7ff202ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:12.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.981+0000 7ff202ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:12.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.981+0000 7ff202ffd640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:42:12.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.981+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2040b0d50 con 0x7ff20414be90 2024-03-25T18:42:12.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2810811414 0 0) 0x7ff1f0003520 con 0x7ff2040b4d60 2024-03-25T18:42:12.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 >> v1:172.21.15.136:6789/0 conn(0x7ff2040b1730 legacy=0x7ff2040b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:12.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 >> v1:172.21.15.32:6789/0 conn(0x7ff20414be90 legacy=0x7ff20414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:12.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:42:12.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 1 monclient: found mon.noname-b 2024-03-25T18:42:12.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:12.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:42:12.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2040b1030 con 0x7ff2040b4d60 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 10 monclient: _finish_auth 0 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7ff1f4001690 con 0x7ff2040b4d60 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff209afb640 5 monclient: authenticate success, global_id 4346 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff209afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff202ffd640 10 monclient: dump: 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:12.991 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:12.992 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:12.992 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:12.992 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:12.992 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:12.989+0000 7ff209afb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:13.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.003+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f40017f0 con 0x7ff2040b4d60 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.003+0000 7ff202ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.003+0000 7ff202ffd640 1 -- 172.21.15.32:0/4230651775 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7ff1f4001c00 con 0x7ff2040b4d60 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.003+0000 7ff202ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.003+0000 7ff202ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.003+0000 7ff202ffd640 10 monclient: dump: 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:13.005 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:13.006 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:13.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.003+0000 7ff209afb640 10 monclient: get_monmap_and_config success 2024-03-25T18:42:13.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.003+0000 7ff209afb640 4 set_mon_vals no callback set 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.004+0000 7ff209afb640 10 monclient: shutdown 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.004+0000 7ff209afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.004+0000 7ff209afb640 1 -- 172.21.15.32:0/4230651775 >> v1:172.21.15.32:6790/0 conn(0x7ff2040b4d60 legacy=0x7ff20413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.004+0000 7ff209afb640 1 -- 172.21.15.32:0/4230651775 shutdown_connections 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.004+0000 7ff209afb640 1 -- 172.21.15.32:0/4230651775 >> 172.21.15.32:0/4230651775 conn(0x7ff2040617a0 msgr2=0x7ff204061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.004+0000 7ff209afb640 1 -- 172.21.15.32:0/4230651775 shutdown_connections 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.004+0000 7ff209afb640 1 -- 172.21.15.32:0/4230651775 wait complete. 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.005+0000 7ff209afb640 10 monclient: build_initial_monmap 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.005+0000 7ff209afb640 10 monclient: monmap: 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:42:13.008 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 librados: starting msgr at 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 librados: starting objecter 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 Processor -- start 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 -- start start 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 librados: setting wanted keys 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 librados: calling monclient init 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 10 monclient: init 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:13.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:42:13.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 10 monclient(hunting): picked mon.noname-c con 0x7ff2040b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:13.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 10 monclient(hunting): picked mon.noname-a con 0x7ff2040b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:13.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 10 monclient(hunting): picked mon.noname-b con 0x7ff20414be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:13.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2041d7980 con 0x7ff2040b4d60 2024-03-25T18:42:13.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2041420a0 con 0x7ff20414be90 2024-03-25T18:42:13.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2041432c0 con 0x7ff2040b1730 2024-03-25T18:42:13.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:13.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.006+0000 7ff209afb640 10 monclient(hunting): authenticate will time out at 6239.196360s 2024-03-25T18:42:13.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.018+0000 7ff203fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7ff20414be90 0x7ff2041d7e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:53928/0 (socket says 172.21.15.32:53928) 2024-03-25T18:42:13.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.018+0000 7ff203fff640 1 -- 172.21.15.32:0/1023491478 learned_addr learned my addr 172.21.15.32:0/1023491478 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:13.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.020+0000 7ff200ff9640 10 client.?.objecter ms_handle_connect 0x7ff2040b1730 2024-03-25T18:42:13.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.020+0000 7ff200ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:13.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.022+0000 7ff200ff9640 10 client.?.objecter ms_handle_connect 0x7ff20414be90 2024-03-25T18:42:13.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.022+0000 7ff200ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:13.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.035+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3891447847 0 0) 0x7ff2041432c0 con 0x7ff2040b1730 2024-03-25T18:42:13.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.035+0000 7ff200ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:42:13.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.035+0000 7ff200ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:13.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.035+0000 7ff200ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:13.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.035+0000 7ff200ff9640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:42:13.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.035+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1e4004eb0 con 0x7ff2040b1730 2024-03-25T18:42:13.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.036+0000 7ff200ff9640 10 client.?.objecter ms_handle_connect 0x7ff2040b4d60 2024-03-25T18:42:13.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.036+0000 7ff200ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:13.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.044+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1662918364 0 0) 0x7ff2041420a0 con 0x7ff20414be90 2024-03-25T18:42:13.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.044+0000 7ff200ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:42:13.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.044+0000 7ff200ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:13.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.044+0000 7ff200ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:13.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.044+0000 7ff200ff9640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:42:13.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.044+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2041432c0 con 0x7ff20414be90 2024-03-25T18:42:13.051 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.050+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3295327438 0 0) 0x7ff2041d7980 con 0x7ff2040b4d60 2024-03-25T18:42:13.051 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.050+0000 7ff200ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:42:13.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.050+0000 7ff200ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:13.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.050+0000 7ff200ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:13.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.050+0000 7ff200ff9640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:42:13.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.050+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff2041420a0 con 0x7ff2040b4d60 2024-03-25T18:42:13.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.064+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1656377774 0 0) 0x7ff1e4004eb0 con 0x7ff2040b1730 2024-03-25T18:42:13.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.064+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2041d7980 con 0x7ff2040b1730 2024-03-25T18:42:13.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.064+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7ff1f4001b00 con 0x7ff2040b1730 2024-03-25T18:42:13.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.064+0000 7ff200ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:13.065 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.064+0000 7ff200ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.064+0000 7ff200ff9640 10 monclient(hunting): dump: 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:13.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.064+0000 7ff200ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:42:13.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.072+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1229810156 0 0) 0x7ff2041420a0 con 0x7ff2040b4d60 2024-03-25T18:42:13.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.072+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1e4004eb0 con 0x7ff2040b4d60 2024-03-25T18:42:13.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.072+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7ff1ec001860 con 0x7ff2040b4d60 2024-03-25T18:42:13.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.072+0000 7ff200ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:13.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.072+0000 7ff200ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.072+0000 7ff200ff9640 10 monclient(hunting): dump: 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:13.074 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.077+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3191455827 0 0) 0x7ff2041432c0 con 0x7ff20414be90 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.077+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff2041420a0 con 0x7ff20414be90 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.077+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7ff1fc001ca0 con 0x7ff20414be90 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.077+0000 7ff200ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.077+0000 7ff200ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.077+0000 7ff200ff9640 10 monclient(hunting): dump: 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:13.078 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:13.079 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:13.079 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:13.079 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:13.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3891636010 0 0) 0x7ff2041d7980 con 0x7ff2040b1730 2024-03-25T18:42:13.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 >> v1:172.21.15.136:6789/0 conn(0x7ff20414be90 legacy=0x7ff2041d7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:13.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 >> v1:172.21.15.32:6789/0 conn(0x7ff2040b4d60 legacy=0x7ff204138360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:13.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:42:13.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 1 monclient: found mon.c 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff204143940 con 0x7ff2040b1730 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 10 monclient: _finish_auth 0 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff200ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 5 monclient: authenticate success, global_id 4349 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 10 monclient: _renew_subs 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff204138a70 con 0x7ff2040b1730 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 10 monclient: _renew_subs 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2041d8700 con 0x7ff2040b1730 2024-03-25T18:42:13.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.085+0000 7ff209afb640 1 librados: init done 2024-03-25T18:42:13.088 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.086+0000 7ff209afb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:13.088 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.086+0000 7ff209afb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:13.088 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.086+0000 7ff209afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:13.088 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.086+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff204143130 con 0x7ff2040b1730 2024-03-25T18:42:13.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.398+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1f4001220 con 0x7ff2040b1730 2024-03-25T18:42:13.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.398+0000 7ff200ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.398+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7ff1f4004c30 con 0x7ff2040b1730 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.398+0000 7ff200ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.398+0000 7ff200ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.398+0000 7ff200ff9640 10 monclient: dump: 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:13.400 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:13.401 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:13.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.398+0000 7ff2027fc640 4 set_mon_vals no callback set 2024-03-25T18:42:13.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.399+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7ff1f401ac00 con 0x7ff2040b1730 2024-03-25T18:42:13.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.400+0000 7ff1fbfff640 10 monclient: get_auth_request con 0x7ff1e406e0f0 auth_method 0 2024-03-25T18:42:13.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.400+0000 7ff200ff9640 10 client.4349.objecter ms_handle_connect 0x7ff1e406e0f0 2024-03-25T18:42:13.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.406+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7ff1f4054870 con 0x7ff2040b1730 2024-03-25T18:42:13.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.406+0000 7ff200ff9640 10 client.4349.objecter ms_dispatch 0x7ff204000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:42:13.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.406+0000 7ff200ff9640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:42:13.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.406+0000 7ff200ff9640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:42:13.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.406+0000 7ff200ff9640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-25T18:42:13.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.406+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1f4050380 con 0x7ff2040b1730 2024-03-25T18:42:13.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.406+0000 7ff200ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:13.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.406+0000 7ff200ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:42:13.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.652+0000 7ff209afb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:13.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.652+0000 7ff209afb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:13.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.652+0000 7ff209afb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:13.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.652+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2041401f0 con 0x7ff2040b1730 2024-03-25T18:42:13.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.686+0000 7ff200ff9640 1 -- 172.21.15.32:0/1023491478 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7ff1f40522b0 con 0x7ff2040b1730 2024-03-25T18:42:13.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.686+0000 7ff200ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:13.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.686+0000 7ff200ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:42:13.687 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:42:13.688 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 10 librados: watch_flush enter 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 10 librados: watch_flush exit 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 10 client.4349.objecter successfully canceled tick 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 >> v1:172.21.15.136:6800/1141502808 conn(0x7ff1e406e0f0 legacy=0x7ff1e40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 10 monclient: shutdown 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 >> v1:172.21.15.32:6790/0 conn(0x7ff2040b1730 legacy=0x7ff204141990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:13.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 shutdown_connections 2024-03-25T18:42:13.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 >> 172.21.15.32:0/1023491478 conn(0x7ff2040617a0 msgr2=0x7ff204061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:13.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.688+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 shutdown_connections 2024-03-25T18:42:13.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.689+0000 7ff209afb640 1 -- 172.21.15.32:0/1023491478 wait complete. 2024-03-25T18:42:13.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:13.689+0000 7ff209afb640 1 librados: shutdown 2024-03-25T18:42:13.702 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:42:20.510 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:42:20.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.627+0000 7f218fbbc640 10 monclient: get_monmap_and_config 2024-03-25T18:42:20.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.627+0000 7f218fbbc640 10 monclient: build_initial_monmap 2024-03-25T18:42:20.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.627+0000 7f218fbbc640 10 monclient: monmap: 2024-03-25T18:42:20.627 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:20.627 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:20.627 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:42:20.627721+0000 2024-03-25T18:42:20.627 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:42:20.627721+0000 2024-03-25T18:42:20.628 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:20.628 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:20.628 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:20.628 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:42:20.628 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:42:20.628 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:20.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.627+0000 7f218fbbc640 1 Processor -- start 2024-03-25T18:42:20.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.628+0000 7f218fbbc640 1 -- start start 2024-03-25T18:42:20.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.628+0000 7f218fbbc640 10 monclient: init 2024-03-25T18:42:20.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.628+0000 7f218fbbc640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:20.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.628+0000 7f218fbbc640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:42:20.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.628+0000 7f218fbbc640 10 monclient(hunting): picked mon.noname-c con 0x7f21880b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:20.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.628+0000 7f218fbbc640 10 monclient(hunting): picked mon.noname-b con 0x7f21880b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:20.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.628+0000 7f218fbbc640 10 monclient(hunting): picked mon.noname-a con 0x7f218814be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:20.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.628+0000 7f218fbbc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21880af940 con 0x7f218814be90 2024-03-25T18:42:20.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.629+0000 7f218fbbc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21880b0a70 con 0x7f21880b4d60 2024-03-25T18:42:20.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.629+0000 7f218fbbc640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21880b0d50 con 0x7f21880b1730 2024-03-25T18:42:20.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.629+0000 7f218fbbc640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:20.630 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.629+0000 7f218fbbc640 10 monclient(hunting): authenticate will time out at 6246.817589s 2024-03-25T18:42:20.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.632+0000 7f218d958640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f21880b4d60 0x7f218813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:55992/0 (socket says 172.21.15.32:55992) 2024-03-25T18:42:20.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.632+0000 7f218d958640 1 -- 172.21.15.32:0/3900129314 learned_addr learned my addr 172.21.15.32:0/3900129314 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:20.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.641+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341359605 0 0) 0x7f21880b0a70 con 0x7f21880b4d60 2024-03-25T18:42:20.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.641+0000 7f218c956640 10 monclient(hunting): _negotiate 2024-03-25T18:42:20.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.641+0000 7f218c956640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:20.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.641+0000 7f218c956640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:20.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.641+0000 7f218c956640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:42:20.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.641+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2174003520 con 0x7f21880b4d60 2024-03-25T18:42:20.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 729460112 0 0) 0x7f2174003520 con 0x7f21880b4d60 2024-03-25T18:42:20.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 >> v1:172.21.15.32:6790/0 conn(0x7f21880b1730 legacy=0x7f21880b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:20.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 >> v1:172.21.15.32:6789/0 conn(0x7f218814be90 legacy=0x7f218814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:20.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 10 monclient: _finish_hunting 0 2024-03-25T18:42:20.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 1 monclient: found mon.noname-c 2024-03-25T18:42:20.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21880b1030 con 0x7f21880b4d60 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 10 monclient: _finish_auth 0 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 10 monclient: _check_auth_tickets 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2184001690 con 0x7f21880b4d60 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218c956640 10 monclient: dump: 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:20.649 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:20.650 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:20.650 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:20.650 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:20.650 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:20.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218fbbc640 5 monclient: authenticate success, global_id 4345 2024-03-25T18:42:20.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.648+0000 7f218fbbc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:20.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21840017f0 con 0x7f21880b4d60 2024-03-25T18:42:20.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218c956640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:20.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218c956640 1 -- 172.21.15.32:0/3900129314 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2184001c40 con 0x7f21880b4d60 2024-03-25T18:42:20.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218c956640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:20.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218c956640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:20.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218c956640 10 monclient: dump: 2024-03-25T18:42:20.659 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:20.659 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218fbbc640 10 monclient: get_monmap_and_config success 2024-03-25T18:42:20.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218fbbc640 4 set_mon_vals no callback set 2024-03-25T18:42:20.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218fbbc640 10 monclient: shutdown 2024-03-25T18:42:20.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218fbbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:20.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.659+0000 7f218fbbc640 1 -- 172.21.15.32:0/3900129314 >> v1:172.21.15.136:6789/0 conn(0x7f21880b4d60 legacy=0x7f218813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:20.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.660+0000 7f218fbbc640 1 -- 172.21.15.32:0/3900129314 shutdown_connections 2024-03-25T18:42:20.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.660+0000 7f218fbbc640 1 -- 172.21.15.32:0/3900129314 >> 172.21.15.32:0/3900129314 conn(0x7f21880617a0 msgr2=0x7f2188061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.660+0000 7f218fbbc640 1 -- 172.21.15.32:0/3900129314 shutdown_connections 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.660+0000 7f218fbbc640 1 -- 172.21.15.32:0/3900129314 wait complete. 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 10 monclient: build_initial_monmap 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 10 monclient: monmap: 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:42:20.663 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 1 librados: starting msgr at 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 1 librados: starting objecter 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 1 Processor -- start 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 1 -- start start 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 1 librados: setting wanted keys 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 1 librados: calling monclient init 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.661+0000 7f218fbbc640 10 monclient: init 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 10 monclient(hunting): picked mon.noname-b con 0x7f21880b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 10 monclient(hunting): picked mon.noname-a con 0x7f21880b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 10 monclient(hunting): picked mon.noname-c con 0x7f218814be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2188141f40 con 0x7f21880b4d60 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2188142360 con 0x7f21880b1730 2024-03-25T18:42:20.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21881424f0 con 0x7f218814be90 2024-03-25T18:42:20.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:20.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.662+0000 7f218fbbc640 10 monclient(hunting): authenticate will time out at 6246.850653s 2024-03-25T18:42:20.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.682+0000 7f218d958640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f21880b1730 0x7f21881405f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:56036/0 (socket says 172.21.15.32:56036) 2024-03-25T18:42:20.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.682+0000 7f218d958640 1 -- 172.21.15.32:0/706069878 learned_addr learned my addr 172.21.15.32:0/706069878 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:20.684 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.684+0000 7f217e7fc640 10 client.?.objecter ms_handle_connect 0x7f21880b1730 2024-03-25T18:42:20.684 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.684+0000 7f217e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:20.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.687+0000 7f217e7fc640 10 client.?.objecter ms_handle_connect 0x7f218814be90 2024-03-25T18:42:20.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.687+0000 7f217e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:20.689 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.690+0000 7f217e7fc640 10 client.?.objecter ms_handle_connect 0x7f21880b4d60 2024-03-25T18:42:20.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.690+0000 7f217e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:20.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.696+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 263874011 0 0) 0x7f2188141f40 con 0x7f21880b4d60 2024-03-25T18:42:20.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.696+0000 7f217e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:42:20.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.696+0000 7f217e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:20.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.696+0000 7f217e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:20.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.696+0000 7f217e7fc640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:42:20.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.696+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f216c004eb0 con 0x7f21880b4d60 2024-03-25T18:42:20.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.703+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2057546508 0 0) 0x7f216c004eb0 con 0x7f21880b4d60 2024-03-25T18:42:20.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.703+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2188141f40 con 0x7f21880b4d60 2024-03-25T18:42:20.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.703+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2178001930 con 0x7f21880b4d60 2024-03-25T18:42:20.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.703+0000 7f217e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:20.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.703+0000 7f217e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:42:20.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.703+0000 7f217e7fc640 10 monclient(hunting): dump: 2024-03-25T18:42:20.705 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:20.705 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:20.705 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:20.705 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:20.705 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:20.705 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:20.706 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:20.706 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:20.706 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:20.706 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:20.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.703+0000 7f217e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:42:20.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.706+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 992122114 0 0) 0x7f2188142360 con 0x7f21880b1730 2024-03-25T18:42:20.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.706+0000 7f217e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:42:20.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.706+0000 7f217e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:20.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.706+0000 7f217e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:20.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.706+0000 7f217e7fc640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:42:20.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.706+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f216c004eb0 con 0x7f21880b1730 2024-03-25T18:42:20.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.710+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3791937979 0 0) 0x7f21880af940 con 0x7f218814be90 2024-03-25T18:42:20.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.710+0000 7f217e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:42:20.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.710+0000 7f217e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:20.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.710+0000 7f217e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:20.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.710+0000 7f217e7fc640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:42:20.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.710+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2188142360 con 0x7f218814be90 2024-03-25T18:42:20.728 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.728+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2082074250 0 0) 0x7f216c004eb0 con 0x7f21880b1730 2024-03-25T18:42:20.728 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.728+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21880af940 con 0x7f21880b1730 2024-03-25T18:42:20.728 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.728+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2184001b00 con 0x7f21880b1730 2024-03-25T18:42:20.728 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.728+0000 7f217e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:20.728 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.728+0000 7f217e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.728+0000 7f217e7fc640 10 monclient(hunting): dump: 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:20.729 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3456589908 0 0) 0x7f2188141f40 con 0x7f21880b4d60 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 >> v1:172.21.15.32:6790/0 conn(0x7f218814be90 legacy=0x7f21881e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 >> v1:172.21.15.136:6789/0 conn(0x7f21880b1730 legacy=0x7f21881405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 1 monclient: found mon.a 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2188140d00 con 0x7f21880b4d60 2024-03-25T18:42:20.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f217e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 10 monclient: _renew_subs 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2188138150 con 0x7f21880b4d60 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 10 monclient: _renew_subs 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21881d8520 con 0x7f21880b4d60 2024-03-25T18:42:20.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.737+0000 7f218fbbc640 1 librados: init done 2024-03-25T18:42:20.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.738+0000 7f218fbbc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:20.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.738+0000 7f218fbbc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:20.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.738+0000 7f218fbbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:20.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:20.738+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2188140f50 con 0x7f21880b4d60 2024-03-25T18:42:21.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.066+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2178003610 con 0x7f21880b4d60 2024-03-25T18:42:21.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.066+0000 7f217e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:21.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.066+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f21780039e0 con 0x7f21880b4d60 2024-03-25T18:42:21.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.066+0000 7f217e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:21.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.066+0000 7f217e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:42:21.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.066+0000 7f217e7fc640 10 monclient: dump: 2024-03-25T18:42:21.066 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:21.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.066+0000 7f217ffff640 4 set_mon_vals no callback set 2024-03-25T18:42:21.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.066+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f2178003c00 con 0x7f21880b4d60 2024-03-25T18:42:21.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.067+0000 7f218d958640 10 monclient: get_auth_request con 0x7f216c06e040 auth_method 0 2024-03-25T18:42:21.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.067+0000 7f217e7fc640 10 client.4365.objecter ms_handle_connect 0x7f216c06e040 2024-03-25T18:42:21.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.072+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f2178053610 con 0x7f21880b4d60 2024-03-25T18:42:21.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.072+0000 7f217e7fc640 10 client.4365.objecter ms_dispatch 0x7f2188000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:42:21.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.072+0000 7f217e7fc640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:42:21.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.072+0000 7f217e7fc640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:42:21.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.072+0000 7f217e7fc640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T18:42:21.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.072+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f217804f120 con 0x7f21880b4d60 2024-03-25T18:42:21.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.072+0000 7f217e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:21.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.072+0000 7f217e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:42:21.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.313+0000 7f218fbbc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:21.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.313+0000 7f218fbbc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:21.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.313+0000 7f218fbbc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:42:21.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.313+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21880627b0 con 0x7f21880b4d60 2024-03-25T18:42:21.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.375+0000 7f217e7fc640 1 -- 172.21.15.32:0/706069878 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f2178051050 con 0x7f21880b4d60 2024-03-25T18:42:21.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.375+0000 7f217e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:21.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.375+0000 7f217e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:42:21.376 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:42:21.376 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 10 librados: watch_flush enter 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 10 librados: watch_flush exit 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 10 client.4365.objecter successfully canceled tick 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 >> v1:172.21.15.136:6800/1141502808 conn(0x7f216c06e040 legacy=0x7f216c070500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 10 monclient: shutdown 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 >> v1:172.21.15.32:6789/0 conn(0x7f21880b4d60 legacy=0x7f21881d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 shutdown_connections 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 >> 172.21.15.32:0/706069878 conn(0x7f21880617a0 msgr2=0x7f21880b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:21.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 shutdown_connections 2024-03-25T18:42:21.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 1 -- 172.21.15.32:0/706069878 wait complete. 2024-03-25T18:42:21.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:21.378+0000 7f218fbbc640 1 librados: shutdown 2024-03-25T18:42:21.389 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:42:27.497 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:39.825 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:27.502 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:39.830 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:27.537 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:39.865 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:27.551 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:39.868 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:27.557 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:39.874 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:27.558 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:03:39.887 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:27.572 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:03:39.900 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:27.586 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:39.903 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:27.604 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:03:39.932 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:28.196 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:42:28.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.311+0000 7f5aeca27640 10 monclient: get_monmap_and_config 2024-03-25T18:42:28.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.311+0000 7f5aeca27640 10 monclient: build_initial_monmap 2024-03-25T18:42:28.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.311+0000 7f5aeca27640 10 monclient: monmap: 2024-03-25T18:42:28.311 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:28.311 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:28.312 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:42:28.311758+0000 2024-03-25T18:42:28.312 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:42:28.311758+0000 2024-03-25T18:42:28.312 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:28.312 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:28.312 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:28.312 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:42:28.312 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:42:28.312 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:28.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.312+0000 7f5aeca27640 1 Processor -- start 2024-03-25T18:42:28.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.312+0000 7f5aeca27640 1 -- start start 2024-03-25T18:42:28.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.312+0000 7f5aeca27640 10 monclient: init 2024-03-25T18:42:28.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.312+0000 7f5aeca27640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:28.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.312+0000 7f5aeca27640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:42:28.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.312+0000 7f5aeca27640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae80b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:28.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.312+0000 7f5aeca27640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae80b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:28.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.312+0000 7f5aeca27640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae814be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:28.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.313+0000 7f5aeca27640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae80af940 con 0x7f5ae80b4d60 2024-03-25T18:42:28.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.313+0000 7f5aeca27640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae80b0a70 con 0x7f5ae814be90 2024-03-25T18:42:28.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.313+0000 7f5aeca27640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae80b0d50 con 0x7f5ae80b1730 2024-03-25T18:42:28.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.313+0000 7f5aeca27640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:28.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.313+0000 7f5aeca27640 10 monclient(hunting): authenticate will time out at 6254.501552s 2024-03-25T18:42:28.323 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.323+0000 7f5ae6d9d640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5ae814be90 0x7f5ae814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:36688/0 (socket says 172.21.15.32:36688) 2024-03-25T18:42:28.323 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.323+0000 7f5ae6d9d640 1 -- 172.21.15.32:0/899589252 learned_addr learned my addr 172.21.15.32:0/899589252 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:28.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.338+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3358850471 0 0) 0x7f5ae80b0a70 con 0x7f5ae814be90 2024-03-25T18:42:28.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.338+0000 7f5ae559a640 10 monclient(hunting): _negotiate 2024-03-25T18:42:28.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.338+0000 7f5ae559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:28.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.338+0000 7f5ae559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:28.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.338+0000 7f5ae559a640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:42:28.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.338+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad0003520 con 0x7f5ae814be90 2024-03-25T18:42:28.342 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.342+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4189097724 0 0) 0x7f5ae80af940 con 0x7f5ae80b4d60 2024-03-25T18:42:28.342 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.342+0000 7f5ae559a640 10 monclient(hunting): _negotiate 2024-03-25T18:42:28.342 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.342+0000 7f5ae559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:28.342 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.342+0000 7f5ae559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:28.342 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.342+0000 7f5ae559a640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:42:28.343 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.342+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae80b0a70 con 0x7f5ae80b4d60 2024-03-25T18:42:28.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 428728609 0 0) 0x7f5ae80b0d50 con 0x7f5ae80b1730 2024-03-25T18:42:28.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 10 monclient(hunting): _negotiate 2024-03-25T18:42:28.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:28.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:28.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:42:28.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ad00039d0 con 0x7f5ae80b1730 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173356696 0 0) 0x7f5ad0003520 con 0x7f5ae814be90 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 >> v1:172.21.15.32:6790/0 conn(0x7f5ae80b1730 legacy=0x7f5ae80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.344+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 >> v1:172.21.15.32:6789/0 conn(0x7f5ae80b4d60 legacy=0x7f5ae813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 10 monclient: _finish_hunting 0 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 1 monclient: found mon.noname-c 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae80b1030 con 0x7f5ae814be90 2024-03-25T18:42:28.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 10 monclient: _finish_auth 0 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 10 monclient: _check_auth_tickets 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5ae0001a10 con 0x7f5ae814be90 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5aeca27640 5 monclient: authenticate success, global_id 4351 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5aeca27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5ae559a640 10 monclient: dump: 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.346 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:28.347 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:28.347 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:28.347 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:28.347 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:28.347 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:28.347 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.345+0000 7f5aeca27640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.359+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ae0001b70 con 0x7f5ae814be90 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.359+0000 7f5ae559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.359+0000 7f5ae559a640 1 -- 172.21.15.32:0/899589252 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5ae0001f80 con 0x7f5ae814be90 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.359+0000 7f5aeca27640 10 monclient: get_monmap_and_config success 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.359+0000 7f5aeca27640 4 set_mon_vals no callback set 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.360+0000 7f5ae559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.360+0000 7f5ae559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.360+0000 7f5ae559a640 10 monclient: dump: 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:28.360 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.361 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.361 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:28.361 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:28.361 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:28.361 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:28.361 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:28.361 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.360+0000 7f5aeca27640 10 monclient: shutdown 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.360+0000 7f5aeca27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.360+0000 7f5aeca27640 1 -- 172.21.15.32:0/899589252 >> v1:172.21.15.136:6789/0 conn(0x7f5ae814be90 legacy=0x7f5ae814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.361+0000 7f5aeca27640 1 -- 172.21.15.32:0/899589252 shutdown_connections 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.361+0000 7f5aeca27640 1 -- 172.21.15.32:0/899589252 >> 172.21.15.32:0/899589252 conn(0x7f5ae80617a0 msgr2=0x7f5ae8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.361+0000 7f5aeca27640 1 -- 172.21.15.32:0/899589252 shutdown_connections 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.361+0000 7f5aeca27640 1 -- 172.21.15.32:0/899589252 wait complete. 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 10 monclient: build_initial_monmap 2024-03-25T18:42:28.363 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 10 monclient: monmap: 2024-03-25T18:42:28.364 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:28.364 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:42:28.364 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:42:28.364 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:42:28.364 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:28.364 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:42:28.364 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 1 librados: starting msgr at 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 1 librados: starting objecter 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 1 Processor -- start 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 1 -- start start 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 1 librados: setting wanted keys 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 1 librados: calling monclient init 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.362+0000 7f5aeca27640 10 monclient: init 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae80b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae80b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae814be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:28.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae8141210 con 0x7f5ae80b4d60 2024-03-25T18:42:28.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae8137640 con 0x7f5ae80b1730 2024-03-25T18:42:28.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae8140380 con 0x7f5ae814be90 2024-03-25T18:42:28.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:28.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.363+0000 7f5aeca27640 10 monclient(hunting): authenticate will time out at 6254.551703s 2024-03-25T18:42:28.370 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.370+0000 7f5ae659c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5ae80b1730 0x7f5ae8143ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:36736/0 (socket says 172.21.15.32:36736) 2024-03-25T18:42:28.370 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.370+0000 7f5ae659c640 1 -- 172.21.15.32:0/2243252227 learned_addr learned my addr 172.21.15.32:0/2243252227 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:28.371 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.371+0000 7f5ad6ffd640 10 client.?.objecter ms_handle_connect 0x7f5ae80b1730 2024-03-25T18:42:28.371 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.371+0000 7f5ad6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:28.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.380+0000 7f5ad6ffd640 10 client.?.objecter ms_handle_connect 0x7f5ae814be90 2024-03-25T18:42:28.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.380+0000 7f5ad6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:28.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.390+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291416798 0 0) 0x7f5ae8137640 con 0x7f5ae80b1730 2024-03-25T18:42:28.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.390+0000 7f5ad6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:28.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.390+0000 7f5ad6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:28.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.390+0000 7f5ad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:28.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.390+0000 7f5ad6ffd640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:42:28.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.390+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac8004eb0 con 0x7f5ae80b1730 2024-03-25T18:42:28.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.391+0000 7f5ad6ffd640 10 client.?.objecter ms_handle_connect 0x7f5ae80b4d60 2024-03-25T18:42:28.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.391+0000 7f5ad6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:28.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.393+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1269054279 0 0) 0x7f5ae8140380 con 0x7f5ae814be90 2024-03-25T18:42:28.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.393+0000 7f5ad6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:28.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.393+0000 7f5ad6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:28.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.393+0000 7f5ad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:28.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.393+0000 7f5ad6ffd640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:42:28.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.393+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae8137640 con 0x7f5ae814be90 2024-03-25T18:42:28.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.405+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755157815 0 0) 0x7f5ae8141210 con 0x7f5ae80b4d60 2024-03-25T18:42:28.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.405+0000 7f5ad6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:28.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.405+0000 7f5ad6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:28.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.405+0000 7f5ad6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:28.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.405+0000 7f5ad6ffd640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:42:28.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.405+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae8140380 con 0x7f5ae80b4d60 2024-03-25T18:42:28.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.414+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4003466749 0 0) 0x7f5ac8004eb0 con 0x7f5ae80b1730 2024-03-25T18:42:28.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.414+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ae8141210 con 0x7f5ae80b1730 2024-03-25T18:42:28.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.414+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5ad8001a70 con 0x7f5ae80b1730 2024-03-25T18:42:28.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.414+0000 7f5ad6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:28.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.414+0000 7f5ad6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:42:28.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.414+0000 7f5ad6ffd640 10 monclient(hunting): dump: 2024-03-25T18:42:28.414 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:28.414 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:28.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.414+0000 7f5ad6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:42:28.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.415+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2218321407 0 0) 0x7f5ae8137640 con 0x7f5ae814be90 2024-03-25T18:42:28.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.415+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ac8004eb0 con 0x7f5ae814be90 2024-03-25T18:42:28.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.415+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5ae0001e80 con 0x7f5ae814be90 2024-03-25T18:42:28.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.415+0000 7f5ad6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:28.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.415+0000 7f5ad6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:42:28.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.415+0000 7f5ad6ffd640 10 monclient(hunting): dump: 2024-03-25T18:42:28.416 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.416+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4164644625 0 0) 0x7f5ae8140380 con 0x7f5ae80b4d60 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.416+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ae8137640 con 0x7f5ae80b4d60 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.416+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5adc001870 con 0x7f5ae80b4d60 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.416+0000 7f5ad6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:28.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.416+0000 7f5ad6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.416+0000 7f5ad6ffd640 10 monclient(hunting): dump: 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:28.418 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:28.419 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:28.419 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2791505576 0 0) 0x7f5ae8141210 con 0x7f5ae80b1730 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 >> v1:172.21.15.32:6790/0 conn(0x7f5ae814be90 legacy=0x7f5ae8141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 >> v1:172.21.15.32:6789/0 conn(0x7f5ae80b4d60 legacy=0x7f5ae8141630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 1 monclient: found mon.b 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae8142450 con 0x7f5ae80b1730 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:42:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5ad6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 5 monclient: authenticate success, global_id 4354 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 10 monclient: _renew_subs 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae8142700 con 0x7f5ae80b1730 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 10 monclient: _renew_subs 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae8137f70 con 0x7f5ae80b1730 2024-03-25T18:42:28.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.424+0000 7f5aeca27640 1 librados: init done 2024-03-25T18:42:28.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.425+0000 7f5aeca27640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:28.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.425+0000 7f5aeca27640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:28.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.425+0000 7f5aeca27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:28.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.425+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae8141050 con 0x7f5ae80b1730 2024-03-25T18:42:28.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.721+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ad8001cd0 con 0x7f5ae80b1730 2024-03-25T18:42:28.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.721+0000 7f5ad6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:28.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.721+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5ad8003a40 con 0x7f5ae80b1730 2024-03-25T18:42:28.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.721+0000 7f5ad6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:28.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.721+0000 7f5ad6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:28.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.721+0000 7f5ad6ffd640 10 monclient: dump: 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:28.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.721+0000 7f5ae4d99640 4 set_mon_vals no callback set 2024-03-25T18:42:28.723 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.722+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5ad8003c60 con 0x7f5ae80b1730 2024-03-25T18:42:28.723 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.723+0000 7f5ae5d9b640 10 monclient: get_auth_request con 0x7f5ac806e140 auth_method 0 2024-03-25T18:42:28.723 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.723+0000 7f5ad6ffd640 10 client.4354.objecter ms_handle_connect 0x7f5ac806e140 2024-03-25T18:42:28.727 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.727+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5ad8054860 con 0x7f5ae80b1730 2024-03-25T18:42:28.727 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.727+0000 7f5ad6ffd640 10 client.4354.objecter ms_dispatch 0x7f5ae8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:42:28.727 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.727+0000 7f5ad6ffd640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:42:28.727 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.727+0000 7f5ad6ffd640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:42:28.727 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.727+0000 7f5ad6ffd640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-25T18:42:28.727 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.727+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5ad8050370 con 0x7f5ae80b1730 2024-03-25T18:42:28.727 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.727+0000 7f5ad6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:28.727 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.727+0000 7f5ad6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:42:28.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.982+0000 7f5aeca27640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:28.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.983+0000 7f5aeca27640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:28.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.983+0000 7f5aeca27640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:28.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:28.983+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ae81377d0 con 0x7f5ae80b1730 2024-03-25T18:42:29.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.016+0000 7f5ad6ffd640 1 -- 172.21.15.32:0/2243252227 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5ad80522a0 con 0x7f5ae80b1730 2024-03-25T18:42:29.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.016+0000 7f5ad6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:29.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.016+0000 7f5ad6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:42:29.017 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:42:29.017 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:42:29.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 10 librados: watch_flush enter 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 10 librados: watch_flush exit 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 10 client.4354.objecter successfully canceled tick 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 >> v1:172.21.15.136:6800/1141502808 conn(0x7f5ac806e140 legacy=0x7f5ac8070600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 10 monclient: shutdown 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 >> v1:172.21.15.136:6789/0 conn(0x7f5ae80b1730 legacy=0x7f5ae8143ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 shutdown_connections 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 >> 172.21.15.32:0/2243252227 conn(0x7f5ae80617a0 msgr2=0x7f5ae80b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 shutdown_connections 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 1 -- 172.21.15.32:0/2243252227 wait complete. 2024-03-25T18:42:29.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:29.019+0000 7f5aeca27640 1 librados: shutdown 2024-03-25T18:42:29.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:42:35.840 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:42:35.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.957+0000 7f34d502a640 10 monclient: get_monmap_and_config 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.957+0000 7f34d502a640 10 monclient: build_initial_monmap 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.957+0000 7f34d502a640 10 monclient: monmap: 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:42:35.958201+0000 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:42:35.958201+0000 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:42:35.958 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.958+0000 7f34d502a640 1 Processor -- start 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.958+0000 7f34d502a640 1 -- start start 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.958+0000 7f34d502a640 10 monclient: init 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 10 monclient(hunting): picked mon.noname-b con 0x7f34d00b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 10 monclient(hunting): picked mon.noname-a con 0x7f34d00b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 10 monclient(hunting): picked mon.noname-c con 0x7f34d014be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d00af940 con 0x7f34d00b1730 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d00b0a70 con 0x7f34d014be90 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d00b0d50 con 0x7f34d00b4d60 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:35.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.959+0000 7f34d502a640 10 monclient(hunting): authenticate will time out at 6262.148283s 2024-03-25T18:42:35.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.972+0000 7f34ced9d640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f34d00b4d60 0x7f34d013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:57686/0 (socket says 172.21.15.32:57686) 2024-03-25T18:42:35.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.972+0000 7f34ced9d640 1 -- 172.21.15.32:0/168106992 learned_addr learned my addr 172.21.15.32:0/168106992 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:35.996 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.996+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4210159046 0 0) 0x7f34d00b0a70 con 0x7f34d014be90 2024-03-25T18:42:35.996 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.996+0000 7f34ce59c640 10 monclient(hunting): _negotiate 2024-03-25T18:42:35.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.996+0000 7f34ce59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:35.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.996+0000 7f34ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:35.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.996+0000 7f34ce59c640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:42:35.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.996+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34bc003520 con 0x7f34d014be90 2024-03-25T18:42:35.998 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.998+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3135158264 0 0) 0x7f34d00b0d50 con 0x7f34d00b4d60 2024-03-25T18:42:35.998 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.998+0000 7f34ce59c640 10 monclient(hunting): _negotiate 2024-03-25T18:42:35.998 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.998+0000 7f34ce59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:35.998 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.998+0000 7f34ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:35.999 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.998+0000 7f34ce59c640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:42:35.999 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:35.998+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d00b0a70 con 0x7f34d00b4d60 2024-03-25T18:42:36.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.005+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10645244 0 0) 0x7f34d00af940 con 0x7f34d00b1730 2024-03-25T18:42:36.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.005+0000 7f34ce59c640 10 monclient(hunting): _negotiate 2024-03-25T18:42:36.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.005+0000 7f34ce59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:36.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.005+0000 7f34ce59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:36.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.005+0000 7f34ce59c640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:42:36.006 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.005+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d00b0d50 con 0x7f34d00b1730 2024-03-25T18:42:36.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3243995685 0 0) 0x7f34bc003520 con 0x7f34d014be90 2024-03-25T18:42:36.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 >> v1:172.21.15.32:6790/0 conn(0x7f34d00b4d60 legacy=0x7f34d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:36.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 >> v1:172.21.15.32:6789/0 conn(0x7f34d00b1730 legacy=0x7f34d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 10 monclient: _finish_hunting 0 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 1 monclient: found mon.noname-c 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d00b1030 con 0x7f34d014be90 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 10 monclient: _finish_auth 0 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 10 monclient: _check_auth_tickets 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f34c8001a10 con 0x7f34d014be90 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34ce59c640 10 monclient: dump: 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:36.014 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34d502a640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:42:36.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.013+0000 7f34d502a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:36.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.020+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34c8001b70 con 0x7f34d014be90 2024-03-25T18:42:36.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.020+0000 7f34ce59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:36.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.020+0000 7f34ce59c640 1 -- 172.21.15.32:0/168106992 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f34c8001f80 con 0x7f34d014be90 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.020+0000 7f34ce59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.020+0000 7f34ce59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.020+0000 7f34ce59c640 10 monclient: dump: 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.020+0000 7f34d502a640 10 monclient: get_monmap_and_config success 2024-03-25T18:42:36.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.020+0000 7f34d502a640 4 set_mon_vals no callback set 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.021+0000 7f34d502a640 10 monclient: shutdown 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.021+0000 7f34d502a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.021+0000 7f34d502a640 1 -- 172.21.15.32:0/168106992 >> v1:172.21.15.136:6789/0 conn(0x7f34d014be90 legacy=0x7f34d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.021+0000 7f34d502a640 1 -- 172.21.15.32:0/168106992 shutdown_connections 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.021+0000 7f34d502a640 1 -- 172.21.15.32:0/168106992 >> 172.21.15.32:0/168106992 conn(0x7f34d00617a0 msgr2=0x7f34d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.021+0000 7f34d502a640 1 -- 172.21.15.32:0/168106992 shutdown_connections 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.021+0000 7f34d502a640 1 -- 172.21.15.32:0/168106992 wait complete. 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.022+0000 7f34d502a640 10 monclient: build_initial_monmap 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.022+0000 7f34d502a640 10 monclient: monmap: 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:36.023 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.023+0000 7f34d502a640 1 librados: starting msgr at 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.023+0000 7f34d502a640 1 librados: starting objecter 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.023+0000 7f34d502a640 1 Processor -- start 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.023+0000 7f34d502a640 1 -- start start 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.023+0000 7f34d502a640 1 librados: setting wanted keys 2024-03-25T18:42:36.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.023+0000 7f34d502a640 1 librados: calling monclient init 2024-03-25T18:42:36.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.023+0000 7f34d502a640 10 monclient: init 2024-03-25T18:42:36.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:36.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:42:36.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 10 monclient(hunting): picked mon.noname-b con 0x7f34d00b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:36.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 10 monclient(hunting): picked mon.noname-c con 0x7f34d00b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:36.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 10 monclient(hunting): picked mon.noname-a con 0x7f34d014be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:36.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d0141d10 con 0x7f34d014be90 2024-03-25T18:42:36.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d0141f80 con 0x7f34d00b1730 2024-03-25T18:42:36.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f34d01425d0 con 0x7f34d00b4d60 2024-03-25T18:42:36.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:36.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.024+0000 7f34d502a640 10 monclient(hunting): authenticate will time out at 6262.212781s 2024-03-25T18:42:36.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.037+0000 7f34c7fff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f34d00b4d60 0x7f34d01d7a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:57734/0 (socket says 172.21.15.32:57734) 2024-03-25T18:42:36.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.037+0000 7f34c7fff640 1 -- 172.21.15.32:0/2213043508 learned_addr learned my addr 172.21.15.32:0/2213043508 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:36.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.040+0000 7f34c77fe640 10 client.?.objecter ms_handle_connect 0x7f34d00b4d60 2024-03-25T18:42:36.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.040+0000 7f34c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:36.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.047+0000 7f34c77fe640 10 client.?.objecter ms_handle_connect 0x7f34d00b1730 2024-03-25T18:42:36.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.047+0000 7f34c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:36.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.055+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290738400 0 0) 0x7f34d0141f80 con 0x7f34d00b1730 2024-03-25T18:42:36.056 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.055+0000 7f34c77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:42:36.056 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.055+0000 7f34c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:36.056 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.055+0000 7f34c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:36.056 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.055+0000 7f34c77fe640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:42:36.056 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.055+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34b0004eb0 con 0x7f34d00b1730 2024-03-25T18:42:36.056 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.056+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119150362 0 0) 0x7f34d01425d0 con 0x7f34d00b4d60 2024-03-25T18:42:36.056 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.056+0000 7f34c77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:42:36.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.056+0000 7f34c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:36.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.056+0000 7f34c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:36.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.056+0000 7f34c77fe640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:42:36.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.056+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d0141f80 con 0x7f34d00b4d60 2024-03-25T18:42:36.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.058+0000 7f34c77fe640 10 client.?.objecter ms_handle_connect 0x7f34d014be90 2024-03-25T18:42:36.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.058+0000 7f34c77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:36.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.069+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2025677726 0 0) 0x7f34b0004eb0 con 0x7f34d00b1730 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.069+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34d01425d0 con 0x7f34d00b1730 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.069+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f34c0001970 con 0x7f34d00b1730 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.069+0000 7f34c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.069+0000 7f34c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.069+0000 7f34c77fe640 10 monclient(hunting): dump: 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:36.070 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:36.071 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:36.071 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:36.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.069+0000 7f34c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:42:36.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.071+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1640614596 0 0) 0x7f34d0141f80 con 0x7f34d00b4d60 2024-03-25T18:42:36.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.071+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f34b0004eb0 con 0x7f34d00b4d60 2024-03-25T18:42:36.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.071+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f34b80019f0 con 0x7f34d00b4d60 2024-03-25T18:42:36.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.071+0000 7f34c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.071+0000 7f34c77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.071+0000 7f34c77fe640 10 monclient(hunting): dump: 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:36.072 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:36.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.073+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4081330880 0 0) 0x7f34d0141d10 con 0x7f34d014be90 2024-03-25T18:42:36.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.073+0000 7f34c77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:42:36.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.073+0000 7f34c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:36.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.073+0000 7f34c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:36.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.073+0000 7f34c77fe640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:42:36.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.073+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f34d0141f80 con 0x7f34d014be90 2024-03-25T18:42:36.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1116689786 0 0) 0x7f34d01425d0 con 0x7f34d00b1730 2024-03-25T18:42:36.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 >> v1:172.21.15.32:6790/0 conn(0x7f34d00b4d60 legacy=0x7f34d01d7a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:36.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 >> v1:172.21.15.32:6789/0 conn(0x7f34d014be90 legacy=0x7f34d01d8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:36.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:42:36.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 1 monclient: found mon.b 2024-03-25T18:42:36.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34d01d8870 con 0x7f34d00b1730 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 10 monclient: _finish_auth 0 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.078+0000 7f34d502a640 5 monclient: authenticate success, global_id 4360 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.079+0000 7f34d502a640 10 monclient: _renew_subs 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.079+0000 7f34d502a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.079+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d0143200 con 0x7f34d00b1730 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.079+0000 7f34d502a640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.079+0000 7f34d502a640 10 monclient: _renew_subs 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.079+0000 7f34d502a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.079+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d0140e60 con 0x7f34d00b1730 2024-03-25T18:42:36.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.079+0000 7f34d502a640 1 librados: init done 2024-03-25T18:42:36.081 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.080+0000 7f34d502a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:36.081 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.080+0000 7f34d502a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:36.081 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.080+0000 7f34d502a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:36.081 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.080+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d00627b0 con 0x7f34d00b1730 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.369+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f34c00047e0 con 0x7f34d00b1730 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.369+0000 7f34c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.369+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f34c0004af0 con 0x7f34d00b1730 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.369+0000 7f34c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.369+0000 7f34c77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.369+0000 7f34c77fe640 10 monclient: dump: 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:36.370 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.371 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:36.371 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:36.371 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:36.371 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:36.371 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:36.371 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:36.371 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:36.371 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.369+0000 7f34cdd9b640 4 set_mon_vals no callback set 2024-03-25T18:42:36.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.370+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f34c001ab00 con 0x7f34d00b1730 2024-03-25T18:42:36.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.371+0000 7f34c7fff640 10 monclient: get_auth_request con 0x7f34b006df50 auth_method 0 2024-03-25T18:42:36.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.371+0000 7f34c77fe640 10 client.4360.objecter ms_handle_connect 0x7f34b006df50 2024-03-25T18:42:36.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.376+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f34c00547c0 con 0x7f34d00b1730 2024-03-25T18:42:36.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.376+0000 7f34c77fe640 10 client.4360.objecter ms_dispatch 0x7f34d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:42:36.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.376+0000 7f34c77fe640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:42:36.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.376+0000 7f34c77fe640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:42:36.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.376+0000 7f34c77fe640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-25T18:42:36.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.376+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f34c0050100 con 0x7f34d00b1730 2024-03-25T18:42:36.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.376+0000 7f34c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:36.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.376+0000 7f34c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:42:36.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.626+0000 7f34d502a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:36.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.626+0000 7f34d502a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:36.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.626+0000 7f34d502a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:36.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.626+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f34d0144830 con 0x7f34d00b1730 2024-03-25T18:42:36.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.654+0000 7f34c77fe640 1 -- 172.21.15.32:0/2213043508 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f34c0052200 con 0x7f34d00b1730 2024-03-25T18:42:36.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.654+0000 7f34c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:36.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.654+0000 7f34c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:42:36.655 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:42:36.655 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:42:36.657 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.656+0000 7f34d502a640 10 librados: watch_flush enter 2024-03-25T18:42:36.657 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.656+0000 7f34d502a640 10 librados: watch_flush exit 2024-03-25T18:42:36.657 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.656+0000 7f34d502a640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.656+0000 7f34d502a640 10 client.4360.objecter successfully canceled tick 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.656+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 >> v1:172.21.15.136:6800/1141502808 conn(0x7f34b006df50 legacy=0x7f34b0070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.656+0000 7f34d502a640 10 monclient: shutdown 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.656+0000 7f34d502a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.656+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 >> v1:172.21.15.136:6789/0 conn(0x7f34d00b1730 legacy=0x7f34d0142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.657+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 shutdown_connections 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.657+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 >> 172.21.15.32:0/2213043508 conn(0x7f34d00617a0 msgr2=0x7f34d00b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.657+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 shutdown_connections 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.657+0000 7f34d502a640 1 -- 172.21.15.32:0/2213043508 wait complete. 2024-03-25T18:42:36.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:36.657+0000 7f34d502a640 1 librados: shutdown 2024-03-25T18:42:36.668 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:42:43.475 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:42:43.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.586+0000 7f2e2c33d640 10 monclient: get_monmap_and_config 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.586+0000 7f2e2c33d640 10 monclient: build_initial_monmap 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.586+0000 7f2e2c33d640 10 monclient: monmap: 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:42:43.588106+0000 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:42:43.588106+0000 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:42:43.588 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:43.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.587+0000 7f2e2c33d640 1 Processor -- start 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.587+0000 7f2e2c33d640 1 -- start start 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.587+0000 7f2e2c33d640 10 monclient: init 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e240b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e240b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e2414be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240af940 con 0x7f2e2414be90 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240b0a70 con 0x7f2e240b4d60 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e240b0d50 con 0x7f2e240b1730 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:43.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.588+0000 7f2e2c33d640 10 monclient(hunting): authenticate will time out at 6269.777976s 2024-03-25T18:42:43.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.597+0000 7f2e2a0d9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2e240b4d60 0x7f2e2413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:57436/0 (socket says 172.21.15.32:57436) 2024-03-25T18:42:43.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.597+0000 7f2e2a0d9640 1 -- 172.21.15.32:0/3928262669 learned_addr learned my addr 172.21.15.32:0/3928262669 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:43.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.609+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2200138068 0 0) 0x7f2e240b0a70 con 0x7f2e240b4d60 2024-03-25T18:42:43.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.609+0000 7f2e290d7640 10 monclient(hunting): _negotiate 2024-03-25T18:42:43.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.609+0000 7f2e290d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:43.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.609+0000 7f2e290d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:43.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.609+0000 7f2e290d7640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:42:43.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.609+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e10003520 con 0x7f2e240b4d60 2024-03-25T18:42:43.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.611+0000 7f2e2a8da640 1 -- 172.21.15.32:0/3928262669 >> v1:172.21.15.32:6789/0 conn(0x7f2e2414be90 legacy=0x7f2e2414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-03-25T18:42:43.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.611+0000 7f2e2a8da640 1 -- 172.21.15.32:0/3928262669 >> v1:172.21.15.32:6789/0 conn(0x7f2e2414be90 legacy=0x7f2e2414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:42:43.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.611+0000 7f2e2a8da640 1 --1- 172.21.15.32:0/3928262669 >> v1:172.21.15.32:6789/0 conn(0x7f2e2414be90 0x7f2e2414e280 :-1 s=OPENED pgs=22347 cs=1 l=1).handle_message read tag failed 2024-03-25T18:42:43.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.611+0000 7f2e2a8da640 1 --1- 172.21.15.32:0/3928262669 >> v1:172.21.15.32:6789/0 conn(0x7f2e2414be90 0x7f2e2414e280 :-1 s=OPENED pgs=22347 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:42:43.613 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.611+0000 7f2e290d7640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.32:6789/0 2024-03-25T18:42:43.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.632+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1031019432 0 0) 0x7f2e240b0d50 con 0x7f2e240b1730 2024-03-25T18:42:43.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.632+0000 7f2e290d7640 10 monclient(hunting): _negotiate 2024-03-25T18:42:43.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.632+0000 7f2e290d7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:43.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.632+0000 7f2e290d7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:43.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.632+0000 7f2e290d7640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:42:43.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.632+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e240b0a70 con 0x7f2e240b1730 2024-03-25T18:42:43.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1873896043 0 0) 0x7f2e10003520 con 0x7f2e240b4d60 2024-03-25T18:42:43.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 >> v1:172.21.15.32:6790/0 conn(0x7f2e240b1730 legacy=0x7f2e240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:43.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 >> v1:172.21.15.32:6789/0 conn(0x7f2e2414be90 legacy=0x7f2e2414e280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:42:43.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 10 monclient: _finish_hunting 0 2024-03-25T18:42:43.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 1 monclient: found mon.noname-c 2024-03-25T18:42:43.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:43.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e240b1030 con 0x7f2e240b4d60 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 10 monclient: _finish_auth 0 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 10 monclient: _check_auth_tickets 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2e20001690 con 0x7f2e240b4d60 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e290d7640 10 monclient: dump: 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.638 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:43.639 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:43.639 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:43.639 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:43.639 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:43.639 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:43.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e2c33d640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:42:43.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.635+0000 7f2e2c33d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:43.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.648+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e200017f0 con 0x7f2e240b4d60 2024-03-25T18:42:43.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.648+0000 7f2e290d7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:43.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.648+0000 7f2e290d7640 1 -- 172.21.15.32:0/3928262669 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2e20001c00 con 0x7f2e240b4d60 2024-03-25T18:42:43.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.648+0000 7f2e290d7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:43.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.648+0000 7f2e290d7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:43.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.648+0000 7f2e290d7640 10 monclient: dump: 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.648+0000 7f2e2c33d640 10 monclient: get_monmap_and_config success 2024-03-25T18:42:43.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.648+0000 7f2e2c33d640 4 set_mon_vals no callback set 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.649+0000 7f2e2c33d640 10 monclient: shutdown 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.649+0000 7f2e2c33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.649+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3928262669 >> v1:172.21.15.136:6789/0 conn(0x7f2e240b4d60 legacy=0x7f2e2413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.649+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3928262669 shutdown_connections 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.649+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3928262669 >> 172.21.15.32:0/3928262669 conn(0x7f2e240617a0 msgr2=0x7f2e24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.649+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3928262669 shutdown_connections 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.649+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3928262669 wait complete. 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 10 monclient: build_initial_monmap 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 10 monclient: monmap: 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:42:43.652 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 1 librados: starting msgr at 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 1 librados: starting objecter 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 1 Processor -- start 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 1 -- start start 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 1 librados: setting wanted keys 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 1 librados: calling monclient init 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.650+0000 7f2e2c33d640 10 monclient: init 2024-03-25T18:42:43.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:43.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:42:43.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 10 monclient(hunting): picked mon.noname-c con 0x7f2e240b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:43.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 10 monclient(hunting): picked mon.noname-a con 0x7f2e240b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:43.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 10 monclient(hunting): picked mon.noname-b con 0x7f2e2414be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:43.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e24145830 con 0x7f2e240b4d60 2024-03-25T18:42:43.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e241444c0 con 0x7f2e2414be90 2024-03-25T18:42:43.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e24144650 con 0x7f2e240b1730 2024-03-25T18:42:43.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:43.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.651+0000 7f2e2c33d640 10 monclient(hunting): authenticate will time out at 6269.841140s 2024-03-25T18:42:43.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.658+0000 7f2e2a8da640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2e2414be90 0x7f2e24142aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:57488/0 (socket says 172.21.15.32:57488) 2024-03-25T18:42:43.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.658+0000 7f2e2a8da640 1 -- 172.21.15.32:0/3503884980 learned_addr learned my addr 172.21.15.32:0/3503884980 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:43.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.659+0000 7f2e1affd640 10 client.?.objecter ms_handle_connect 0x7f2e2414be90 2024-03-25T18:42:43.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.659+0000 7f2e1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:43.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.659+0000 7f2e1affd640 10 client.?.objecter ms_handle_connect 0x7f2e240b4d60 2024-03-25T18:42:43.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.659+0000 7f2e1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:43.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.661+0000 7f2e1affd640 10 client.?.objecter ms_handle_connect 0x7f2e240b1730 2024-03-25T18:42:43.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.661+0000 7f2e1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:43.667 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.666+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2693256416 0 0) 0x7f2e24145830 con 0x7f2e240b4d60 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.666+0000 7f2e1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.666+0000 7f2e1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.666+0000 7f2e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.666+0000 7f2e1affd640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.666+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e08004eb0 con 0x7f2e240b4d60 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.667+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2519187004 0 0) 0x7f2e241444c0 con 0x7f2e2414be90 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.667+0000 7f2e1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.667+0000 7f2e1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.667+0000 7f2e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.667+0000 7f2e1affd640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:42:43.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.667+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e24145830 con 0x7f2e2414be90 2024-03-25T18:42:43.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.678+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106078328 0 0) 0x7f2e24144650 con 0x7f2e240b1730 2024-03-25T18:42:43.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.678+0000 7f2e1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:43.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.678+0000 7f2e1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:43.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.678+0000 7f2e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:43.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.678+0000 7f2e1affd640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:42:43.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.678+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e241444c0 con 0x7f2e240b1730 2024-03-25T18:42:43.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.690+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1549667728 0 0) 0x7f2e08004eb0 con 0x7f2e240b4d60 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.690+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e24144650 con 0x7f2e240b4d60 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.690+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2e14001930 con 0x7f2e240b4d60 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.690+0000 7f2e1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.690+0000 7f2e1affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.690+0000 7f2e1affd640 10 monclient(hunting): dump: 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:43.692 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:43.693 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:43.693 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:43.693 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:43.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.690+0000 7f2e1affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.694+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 424936288 0 0) 0x7f2e241444c0 con 0x7f2e240b1730 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.694+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e08004eb0 con 0x7f2e240b1730 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.694+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2e20001b00 con 0x7f2e240b1730 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.694+0000 7f2e1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.694+0000 7f2e1affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.694+0000 7f2e1affd640 10 monclient(hunting): dump: 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:43.696 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:43.697 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:43.697 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:43.697 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:43.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.696+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 36268880 0 0) 0x7f2e24145830 con 0x7f2e2414be90 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.696+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e241444c0 con 0x7f2e2414be90 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.696+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2e1c001c00 con 0x7f2e2414be90 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.696+0000 7f2e1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.696+0000 7f2e1affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.696+0000 7f2e1affd640 10 monclient(hunting): dump: 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:43.698 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:43.699 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:43.699 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:43.699 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:43.699 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:43.699 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.706+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4238290691 0 0) 0x7f2e08004eb0 con 0x7f2e240b1730 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.706+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 >> v1:172.21.15.136:6789/0 conn(0x7f2e2414be90 legacy=0x7f2e24142aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.706+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 >> v1:172.21.15.32:6789/0 conn(0x7f2e240b4d60 legacy=0x7f2e2413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.706+0000 7f2e1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e1affd640 1 monclient: found mon.c 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e1affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e241431b0 con 0x7f2e240b1730 2024-03-25T18:42:43.711 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e1affd640 10 monclient: _finish_auth 0 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 5 monclient: authenticate success, global_id 4373 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 10 monclient: _renew_subs 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e24143460 con 0x7f2e240b1730 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 10 monclient: _renew_subs 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:43.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e241320a0 con 0x7f2e240b1730 2024-03-25T18:42:43.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.707+0000 7f2e2c33d640 1 librados: init done 2024-03-25T18:42:43.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.708+0000 7f2e2c33d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:43.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.708+0000 7f2e2c33d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:43.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.708+0000 7f2e2c33d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:43.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:43.708+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e240627b0 con 0x7f2e240b1730 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.028+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e20001c60 con 0x7f2e240b1730 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.028+0000 7f2e1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.028+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2e20004c30 con 0x7f2e240b1730 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.028+0000 7f2e1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.028+0000 7f2e1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.028+0000 7f2e1affd640 10 monclient: dump: 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:44.029 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:44.030 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:44.030 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:44.030 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:44.030 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:44.030 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:44.030 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:44.030 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:44.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.028+0000 7f2e288d6640 4 set_mon_vals no callback set 2024-03-25T18:42:44.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.028+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f2e2001ac00 con 0x7f2e240b1730 2024-03-25T18:42:44.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.029+0000 7f2e298d8640 10 monclient: get_auth_request con 0x7f2e0806e100 auth_method 0 2024-03-25T18:42:44.031 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.029+0000 7f2e1affd640 10 client.4373.objecter ms_handle_connect 0x7f2e0806e100 2024-03-25T18:42:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.036+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f2e20054800 con 0x7f2e240b1730 2024-03-25T18:42:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.036+0000 7f2e1affd640 10 client.4373.objecter ms_dispatch 0x7f2e24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:42:44.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.036+0000 7f2e1affd640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:42:44.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.036+0000 7f2e1affd640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:42:44.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.036+0000 7f2e1affd640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-25T18:42:44.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.036+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e20050310 con 0x7f2e240b1730 2024-03-25T18:42:44.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.036+0000 7f2e1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:44.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.036+0000 7f2e1affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:42:44.277 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.276+0000 7f2e2c33d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:44.277 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.276+0000 7f2e2c33d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:44.277 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.276+0000 7f2e2c33d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:42:44.277 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.276+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e24069f40 con 0x7f2e240b1730 2024-03-25T18:42:44.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.374+0000 7f2e1affd640 1 -- 172.21.15.32:0/3503884980 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f2e20052240 con 0x7f2e240b1730 2024-03-25T18:42:44.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.374+0000 7f2e1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:44.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.374+0000 7f2e1affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:42:44.376 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:42:44.376 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:42:44.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 10 librados: watch_flush enter 2024-03-25T18:42:44.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 10 librados: watch_flush exit 2024-03-25T18:42:44.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-25T18:42:44.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 10 client.4373.objecter successfully canceled tick 2024-03-25T18:42:44.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 >> v1:172.21.15.136:6800/1141502808 conn(0x7f2e0806e100 legacy=0x7f2e080705c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:44.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 10 monclient: shutdown 2024-03-25T18:42:44.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:44.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 >> v1:172.21.15.32:6790/0 conn(0x7f2e240b1730 legacy=0x7f2e24143d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:44.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 shutdown_connections 2024-03-25T18:42:44.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 >> 172.21.15.32:0/3503884980 conn(0x7f2e240617a0 msgr2=0x7f2e2413c550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:44.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 shutdown_connections 2024-03-25T18:42:44.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 1 -- 172.21.15.32:0/3503884980 wait complete. 2024-03-25T18:42:44.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:44.376+0000 7f2e2c33d640 1 librados: shutdown 2024-03-25T18:42:44.390 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:42:47.570 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:59.898 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:47.573 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:59.901 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:47.610 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:03:59.938 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:47.633 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:03:59.961 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:47.639 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:59.956 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:47.643 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:59.960 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:47.643 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:03:59.971 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:47.677 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:04:00.005 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:47.681 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:03:59.998 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:42:51.198 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:42:51.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.310+0000 7f098d805640 10 monclient: get_monmap_and_config 2024-03-25T18:42:51.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.310+0000 7f098d805640 10 monclient: build_initial_monmap 2024-03-25T18:42:51.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.310+0000 7f098d805640 10 monclient: monmap: 2024-03-25T18:42:51.311 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:42:51.311999+0000 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:42:51.311999+0000 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:42:51.312 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:51.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 1 Processor -- start 2024-03-25T18:42:51.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 1 -- start start 2024-03-25T18:42:51.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 10 monclient: init 2024-03-25T18:42:51.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:51.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:42:51.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 10 monclient(hunting): picked mon.noname-a con 0x7f09880a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:51.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 10 monclient(hunting): picked mon.noname-b con 0x7f098813f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:51.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 10 monclient(hunting): picked mon.noname-c con 0x7f09881430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:51.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0988142520 con 0x7f09880a98f0 2024-03-25T18:42:51.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880b28a0 con 0x7f09881430c0 2024-03-25T18:42:51.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880b2a30 con 0x7f098813f710 2024-03-25T18:42:51.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:51.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.311+0000 7f098d805640 10 monclient(hunting): authenticate will time out at 6277.501483s 2024-03-25T18:42:51.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.343+0000 7f09877fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f09881430c0 0x7f098814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:35776/0 (socket says 172.21.15.32:35776) 2024-03-25T18:42:51.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.343+0000 7f09877fe640 1 -- 172.21.15.32:0/2044208353 learned_addr learned my addr 172.21.15.32:0/2044208353 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:51.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.365+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3163173552 0 0) 0x7f09880b28a0 con 0x7f09881430c0 2024-03-25T18:42:51.367 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.365+0000 7f09867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:42:51.367 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.365+0000 7f09867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:51.367 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.365+0000 7f09867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:51.367 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.365+0000 7f09867fc640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:42:51.367 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.365+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0974003520 con 0x7f09881430c0 2024-03-25T18:42:51.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1805708021 0 0) 0x7f0974003520 con 0x7f09881430c0 2024-03-25T18:42:51.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 >> v1:172.21.15.32:6790/0 conn(0x7f098813f710 legacy=0x7f09881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:51.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 >> v1:172.21.15.32:6789/0 conn(0x7f09880a98f0 legacy=0x7f09880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:51.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:42:51.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 1 monclient: found mon.noname-c 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09880b44c0 con 0x7f09881430c0 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 10 monclient: _finish_auth 0 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0980001a10 con 0x7f09881430c0 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f09867fc640 10 monclient: dump: 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:51.376 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:51.377 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:51.377 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:51.377 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:51.377 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:51.377 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:51.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f098d805640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:42:51.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.374+0000 7f098d805640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:51.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.389+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0980001b70 con 0x7f09881430c0 2024-03-25T18:42:51.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.389+0000 7f09867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:51.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.389+0000 7f09867fc640 1 -- 172.21.15.32:0/2044208353 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0980001f80 con 0x7f09881430c0 2024-03-25T18:42:51.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.389+0000 7f09867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:51.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.389+0000 7f09867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.389+0000 7f09867fc640 10 monclient: dump: 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.389+0000 7f098d805640 10 monclient: get_monmap_and_config success 2024-03-25T18:42:51.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.389+0000 7f098d805640 4 set_mon_vals no callback set 2024-03-25T18:42:51.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.390+0000 7f098d805640 10 monclient: shutdown 2024-03-25T18:42:51.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.390+0000 7f098d805640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.390+0000 7f098d805640 1 -- 172.21.15.32:0/2044208353 >> v1:172.21.15.136:6789/0 conn(0x7f09881430c0 legacy=0x7f098814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.390+0000 7f098d805640 1 -- 172.21.15.32:0/2044208353 shutdown_connections 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.390+0000 7f098d805640 1 -- 172.21.15.32:0/2044208353 >> 172.21.15.32:0/2044208353 conn(0x7f09880b7ea0 msgr2=0x7f09880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.390+0000 7f098d805640 1 -- 172.21.15.32:0/2044208353 shutdown_connections 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.390+0000 7f098d805640 1 -- 172.21.15.32:0/2044208353 wait complete. 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 10 monclient: build_initial_monmap 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 10 monclient: monmap: 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:51.394 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 1 librados: starting msgr at 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 1 librados: starting objecter 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 1 Processor -- start 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 1 -- start start 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 1 librados: setting wanted keys 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 1 librados: calling monclient init 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.391+0000 7f098d805640 10 monclient: init 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:42:51.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 10 monclient(hunting): picked mon.noname-b con 0x7f09880a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 10 monclient(hunting): picked mon.noname-a con 0x7f098813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 10 monclient(hunting): picked mon.noname-c con 0x7f09881430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0988136ad0 con 0x7f098813f710 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09880b7520 con 0x7f09880a98f0 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0988135810 con 0x7f09881430c0 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.392+0000 7f098d805640 10 monclient(hunting): authenticate will time out at 6277.582114s 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.394+0000 7f0986ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f09880a98f0 0x7f0988135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:35812/0 (socket says 172.21.15.32:35812) 2024-03-25T18:42:51.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.394+0000 7f0986ffd640 1 -- 172.21.15.32:0/3535380855 learned_addr learned my addr 172.21.15.32:0/3535380855 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:51.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.394+0000 7f0984ff9640 10 client.?.objecter ms_handle_connect 0x7f09880a98f0 2024-03-25T18:42:51.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.394+0000 7f0984ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:51.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.399+0000 7f0984ff9640 10 client.?.objecter ms_handle_connect 0x7f09881430c0 2024-03-25T18:42:51.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.399+0000 7f0984ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:51.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.400+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3682489804 0 0) 0x7f09880b7520 con 0x7f09880a98f0 2024-03-25T18:42:51.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.400+0000 7f0984ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:42:51.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.400+0000 7f0984ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:51.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.400+0000 7f0984ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:51.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.400+0000 7f0984ff9640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:42:51.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.400+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0968004eb0 con 0x7f09880a98f0 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4110097663 0 0) 0x7f0968004eb0 con 0x7f09880a98f0 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09880b7520 con 0x7f09880a98f0 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0978001a30 con 0x7f09880a98f0 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 10 monclient(hunting): dump: 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:51.410 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1924193050 0 0) 0x7f0988135810 con 0x7f09881430c0 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:42:51.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.408+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0968004eb0 con 0x7f09881430c0 2024-03-25T18:42:51.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.410+0000 7f0984ff9640 10 client.?.objecter ms_handle_connect 0x7f098813f710 2024-03-25T18:42:51.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.410+0000 7f0984ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:51.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1506527272 0 0) 0x7f09880b7520 con 0x7f09880a98f0 2024-03-25T18:42:51.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 >> v1:172.21.15.32:6790/0 conn(0x7f09881430c0 legacy=0x7f0988130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:51.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 >> v1:172.21.15.32:6789/0 conn(0x7f098813f710 legacy=0x7f09880b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:51.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:42:51.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 1 monclient: found mon.b 2024-03-25T18:42:51.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:51.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09880b6180 con 0x7f09880a98f0 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 10 monclient: _finish_auth 0 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f0984ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 5 monclient: authenticate success, global_id 4372 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 10 monclient: _renew_subs 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09880b7a40 con 0x7f09880a98f0 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 10 client.4372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 10 monclient: _renew_subs 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0988131840 con 0x7f09880a98f0 2024-03-25T18:42:51.417 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.415+0000 7f098d805640 1 librados: init done 2024-03-25T18:42:51.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.416+0000 7f098d805640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:51.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.416+0000 7f098d805640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:51.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.416+0000 7f098d805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:51.418 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.416+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0988136f60 con 0x7f09880a98f0 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.685+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0978001cf0 con 0x7f09880a98f0 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.685+0000 7f0984ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.685+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f09780038c0 con 0x7f09880a98f0 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.685+0000 7f0984ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.685+0000 7f0984ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.685+0000 7f0984ff9640 10 monclient: dump: 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:51.687 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:51.688 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:51.688 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:51.688 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:51.688 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:51.688 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:51.688 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:51.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.685+0000 7f0985ffb640 4 set_mon_vals no callback set 2024-03-25T18:42:51.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.686+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f0978003ae0 con 0x7f09880a98f0 2024-03-25T18:42:51.689 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.687+0000 7f097ffff640 10 monclient: get_auth_request con 0x7f096806dd90 auth_method 0 2024-03-25T18:42:51.689 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.687+0000 7f0984ff9640 10 client.4372.objecter ms_handle_connect 0x7f096806dd90 2024-03-25T18:42:51.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.691+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f0978054710 con 0x7f09880a98f0 2024-03-25T18:42:51.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.691+0000 7f0984ff9640 10 client.4372.objecter ms_dispatch 0x7f0988000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:42:51.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.691+0000 7f0984ff9640 3 client.4372.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:42:51.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.691+0000 7f0984ff9640 3 client.4372.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:42:51.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.691+0000 7f0984ff9640 20 client.4372.objecter dump_active .. 0 homeless 2024-03-25T18:42:51.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.691+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0978050220 con 0x7f09880a98f0 2024-03-25T18:42:51.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.691+0000 7f0984ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:51.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.691+0000 7f0984ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:42:51.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.949+0000 7f098d805640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:51.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.949+0000 7f098d805640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:51.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.949+0000 7f098d805640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:51.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.949+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09880b6310 con 0x7f09880a98f0 2024-03-25T18:42:51.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.976+0000 7f0984ff9640 1 -- 172.21.15.32:0/3535380855 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f0978052150 con 0x7f09880a98f0 2024-03-25T18:42:51.978 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.976+0000 7f0984ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:51.978 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.976+0000 7f0984ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:42:51.978 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:42:51.978 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 10 librados: watch_flush enter 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 10 librados: watch_flush exit 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 20 client.4372.objecter shutdown clearing up homeless session... 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 10 client.4372.objecter successfully canceled tick 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 >> v1:172.21.15.136:6800/1141502808 conn(0x7f096806dd90 legacy=0x7f0968070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 10 monclient: shutdown 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 >> v1:172.21.15.136:6789/0 conn(0x7f09880a98f0 legacy=0x7f0988135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 shutdown_connections 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 >> 172.21.15.32:0/3535380855 conn(0x7f09880b7ea0 msgr2=0x7f0988074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:51.980 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 shutdown_connections 2024-03-25T18:42:51.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.978+0000 7f098d805640 1 -- 172.21.15.32:0/3535380855 wait complete. 2024-03-25T18:42:51.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:51.979+0000 7f098d805640 1 librados: shutdown 2024-03-25T18:42:51.991 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:42:58.799 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:42:58.915 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.914+0000 7f58ea486640 10 monclient: get_monmap_and_config 2024-03-25T18:42:58.915 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.914+0000 7f58ea486640 10 monclient: build_initial_monmap 2024-03-25T18:42:58.915 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.914+0000 7f58ea486640 10 monclient: monmap: 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:42:58.916028+0000 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:42:58.916028+0000 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:42:58.916 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.915+0000 7f58ea486640 1 Processor -- start 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.915+0000 7f58ea486640 1 -- start start 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.915+0000 7f58ea486640 10 monclient: init 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 10 monclient(hunting): picked mon.noname-a con 0x7f58e40a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 10 monclient(hunting): picked mon.noname-c con 0x7f58e413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 10 monclient(hunting): picked mon.noname-b con 0x7f58e41430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e4142520 con 0x7f58e40a98f0 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40b28a0 con 0x7f58e413f710 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e40b2a30 con 0x7f58e41430c0 2024-03-25T18:42:58.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:58.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.916+0000 7f58ea486640 10 monclient(hunting): authenticate will time out at 6285.105843s 2024-03-25T18:42:58.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.931+0000 7f58e37fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f58e413f710 0x7f58e41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:42272/0 (socket says 172.21.15.32:42272) 2024-03-25T18:42:58.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.931+0000 7f58e37fe640 1 -- 172.21.15.32:0/3678335162 learned_addr learned my addr 172.21.15.32:0/3678335162 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:58.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.941+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553028112 0 0) 0x7f58e40b28a0 con 0x7f58e413f710 2024-03-25T18:42:58.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.941+0000 7f58e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:42:58.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.941+0000 7f58e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:58.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.941+0000 7f58e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:58.942 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.941+0000 7f58e2ffd640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:42:58.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.941+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58cc003520 con 0x7f58e413f710 2024-03-25T18:42:58.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1178176978 0 0) 0x7f58cc003520 con 0x7f58e413f710 2024-03-25T18:42:58.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 >> v1:172.21.15.32:6790/0 conn(0x7f58e41430c0 legacy=0x7f58e414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 >> v1:172.21.15.32:6789/0 conn(0x7f58e40a98f0 legacy=0x7f58e40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e40b44c0 con 0x7f58e413f710 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f58d8001690 con 0x7f58e413f710 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58e2ffd640 10 monclient: dump: 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:58.962 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58ea486640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:42:58.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.960+0000 7f58ea486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d80017f0 con 0x7f58e413f710 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58e2ffd640 1 -- 172.21.15.32:0/3678335162 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f58d8001c40 con 0x7f58e413f710 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58ea486640 10 monclient: get_monmap_and_config success 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58ea486640 4 set_mon_vals no callback set 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58e2ffd640 10 monclient: dump: 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:58.970 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:58.971 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:58.971 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:58.971 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:58.971 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:58.971 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:58.971 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:58.971 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:58.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58ea486640 10 monclient: shutdown 2024-03-25T18:42:58.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58ea486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:58.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58ea486640 1 -- 172.21.15.32:0/3678335162 >> v1:172.21.15.136:6789/0 conn(0x7f58e413f710 legacy=0x7f58e41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:58.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58ea486640 1 -- 172.21.15.32:0/3678335162 shutdown_connections 2024-03-25T18:42:58.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.969+0000 7f58ea486640 1 -- 172.21.15.32:0/3678335162 >> 172.21.15.32:0/3678335162 conn(0x7f58e40b7ea0 msgr2=0x7f58e40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.970+0000 7f58ea486640 1 -- 172.21.15.32:0/3678335162 shutdown_connections 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.970+0000 7f58ea486640 1 -- 172.21.15.32:0/3678335162 wait complete. 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.970+0000 7f58ea486640 10 monclient: build_initial_monmap 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.970+0000 7f58ea486640 10 monclient: monmap: 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:58.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.971+0000 7f58ea486640 1 librados: starting msgr at 2024-03-25T18:42:58.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.971+0000 7f58ea486640 1 librados: starting objecter 2024-03-25T18:42:58.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.971+0000 7f58ea486640 1 Processor -- start 2024-03-25T18:42:58.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.971+0000 7f58ea486640 1 -- start start 2024-03-25T18:42:58.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.971+0000 7f58ea486640 1 librados: setting wanted keys 2024-03-25T18:42:58.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.971+0000 7f58ea486640 1 librados: calling monclient init 2024-03-25T18:42:58.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.971+0000 7f58ea486640 10 monclient: init 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 10 monclient: _reopen_session rank -1 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 10 monclient(hunting): picked mon.noname-b con 0x7f58e40a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 10 monclient(hunting): picked mon.noname-c con 0x7f58e413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 10 monclient(hunting): picked mon.noname-a con 0x7f58e41430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e4137df0 con 0x7f58e41430c0 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e41337a0 con 0x7f58e40a98f0 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58e4133930 con 0x7f58e413f710 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 10 monclient(hunting): _renew_subs 2024-03-25T18:42:58.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.972+0000 7f58ea486640 10 monclient(hunting): authenticate will time out at 6285.161917s 2024-03-25T18:42:58.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.981+0000 7f58e3fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f58e40a98f0 0x7f58e41392c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:42298/0 (socket says 172.21.15.32:42298) 2024-03-25T18:42:58.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.981+0000 7f58e3fff640 1 -- 172.21.15.32:0/2008095832 learned_addr learned my addr 172.21.15.32:0/2008095832 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:42:58.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.982+0000 7f58e0ff9640 10 client.?.objecter ms_handle_connect 0x7f58e40a98f0 2024-03-25T18:42:58.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.982+0000 7f58e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:58.992 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.991+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553733494 0 0) 0x7f58e41337a0 con 0x7f58e40a98f0 2024-03-25T18:42:58.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.991+0000 7f58e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:42:58.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.991+0000 7f58e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:58.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.991+0000 7f58e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:58.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.991+0000 7f58e0ff9640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:42:58.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.991+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d0004eb0 con 0x7f58e40a98f0 2024-03-25T18:42:58.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.996+0000 7f58e0ff9640 10 client.?.objecter ms_handle_connect 0x7f58e413f710 2024-03-25T18:42:58.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:58.996+0000 7f58e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:59.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.002+0000 7f58e0ff9640 10 client.?.objecter ms_handle_connect 0x7f58e41430c0 2024-03-25T18:42:59.003 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.002+0000 7f58e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.007+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1629781533 0 0) 0x7f58d0004eb0 con 0x7f58e40a98f0 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.007+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58e41337a0 con 0x7f58e40a98f0 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.007+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f58d4001a30 con 0x7f58e40a98f0 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.007+0000 7f58e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.007+0000 7f58e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.007+0000 7f58e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:59.008 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:59.009 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:59.009 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:59.009 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:59.009 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:59.009 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:59.009 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:59.009 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:59.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.007+0000 7f58e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:42:59.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.010+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3308000745 0 0) 0x7f58e4133930 con 0x7f58e413f710 2024-03-25T18:42:59.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.010+0000 7f58e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:42:59.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.010+0000 7f58e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:59.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.010+0000 7f58e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:59.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.010+0000 7f58e0ff9640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:42:59.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.010+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58d0004eb0 con 0x7f58e413f710 2024-03-25T18:42:59.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.016+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3433584993 0 0) 0x7f58e4137df0 con 0x7f58e41430c0 2024-03-25T18:42:59.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.016+0000 7f58e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:42:59.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.016+0000 7f58e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:42:59.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.016+0000 7f58e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:42:59.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.016+0000 7f58e0ff9640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:42:59.018 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.016+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58e4133930 con 0x7f58e41430c0 2024-03-25T18:42:59.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2640334182 0 0) 0x7f58e41337a0 con 0x7f58e40a98f0 2024-03-25T18:42:59.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 >> v1:172.21.15.32:6790/0 conn(0x7f58e413f710 legacy=0x7f58e41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 >> v1:172.21.15.32:6789/0 conn(0x7f58e41430c0 legacy=0x7f58e4133090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 1 monclient: found mon.b 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e41367d0 con 0x7f58e40a98f0 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 10 monclient: _renew_subs 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58e41369e0 con 0x7f58e40a98f0 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 10 client.4378.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:42:59.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 10 monclient: _renew_subs 2024-03-25T18:42:59.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:59.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58e413b640 con 0x7f58e40a98f0 2024-03-25T18:42:59.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.019+0000 7f58ea486640 1 librados: init done 2024-03-25T18:42:59.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.020+0000 7f58ea486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:59.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.020+0000 7f58ea486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:59.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.020+0000 7f58ea486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:59.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.020+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58e40627b0 con 0x7f58e40a98f0 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.286+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58d4001cb0 con 0x7f58e40a98f0 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.286+0000 7f58e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.286+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f58d4003a60 con 0x7f58e40a98f0 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.286+0000 7f58e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.286+0000 7f58e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.286+0000 7f58e0ff9640 10 monclient: dump: 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:42:59.288 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.286+0000 7f58e27fc640 4 set_mon_vals no callback set 2024-03-25T18:42:59.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.287+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f58d4003c80 con 0x7f58e40a98f0 2024-03-25T18:42:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.288+0000 7f58e37fe640 10 monclient: get_auth_request con 0x7f58d006de10 auth_method 0 2024-03-25T18:42:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.288+0000 7f58e0ff9640 10 client.4378.objecter ms_handle_connect 0x7f58d006de10 2024-03-25T18:42:59.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.293+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f58d40548d0 con 0x7f58e40a98f0 2024-03-25T18:42:59.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.293+0000 7f58e0ff9640 10 client.4378.objecter ms_dispatch 0x7f58e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:42:59.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.293+0000 7f58e0ff9640 3 client.4378.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:42:59.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.293+0000 7f58e0ff9640 3 client.4378.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:42:59.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.293+0000 7f58e0ff9640 20 client.4378.objecter dump_active .. 0 homeless 2024-03-25T18:42:59.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.294+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f58d40503e0 con 0x7f58e40a98f0 2024-03-25T18:42:59.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.294+0000 7f58e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:42:59.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.294+0000 7f58e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:42:59.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.540+0000 7f58ea486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:59.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.540+0000 7f58ea486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:59.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.540+0000 7f58ea486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:42:59.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.540+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58e4137300 con 0x7f58e40a98f0 2024-03-25T18:42:59.567 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.566+0000 7f58e0ff9640 1 -- 172.21.15.32:0/2008095832 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f58d4052310 con 0x7f58e40a98f0 2024-03-25T18:42:59.567 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.566+0000 7f58e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:42:59.567 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.566+0000 7f58e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:42:59.567 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:42:59.568 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:42:59.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 10 librados: watch_flush enter 2024-03-25T18:42:59.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 10 librados: watch_flush exit 2024-03-25T18:42:59.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 20 client.4378.objecter shutdown clearing up homeless session... 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 10 client.4378.objecter successfully canceled tick 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 >> v1:172.21.15.136:6800/1141502808 conn(0x7f58d006de10 legacy=0x7f58d00702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 10 monclient: shutdown 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 >> v1:172.21.15.136:6789/0 conn(0x7f58e40a98f0 legacy=0x7f58e41392c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 shutdown_connections 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 >> 172.21.15.32:0/2008095832 conn(0x7f58e40b7ea0 msgr2=0x7f58e4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 shutdown_connections 2024-03-25T18:42:59.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.568+0000 7f58ea486640 1 -- 172.21.15.32:0/2008095832 wait complete. 2024-03-25T18:42:59.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:42:59.569+0000 7f58ea486640 1 librados: shutdown 2024-03-25T18:42:59.581 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:43:06.386 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:43:06.507 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.507+0000 7f57d6ec9640 10 monclient: get_monmap_and_config 2024-03-25T18:43:06.507 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.507+0000 7f57d6ec9640 10 monclient: build_initial_monmap 2024-03-25T18:43:06.507 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.507+0000 7f57d6ec9640 10 monclient: monmap: 2024-03-25T18:43:06.507 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:06.507 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:06.508 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:43:06.507958+0000 2024-03-25T18:43:06.508 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:43:06.507958+0000 2024-03-25T18:43:06.508 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:06.508 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:06.508 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:06.508 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:43:06.508 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:43:06.508 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:06.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.508+0000 7f57d6ec9640 1 Processor -- start 2024-03-25T18:43:06.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.508+0000 7f57d6ec9640 1 -- start start 2024-03-25T18:43:06.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.508+0000 7f57d6ec9640 10 monclient: init 2024-03-25T18:43:06.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.508+0000 7f57d6ec9640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:06.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.508+0000 7f57d6ec9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:43:06.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.509+0000 7f57d6ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f57d006ab80 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:06.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.509+0000 7f57d6ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f57d006af60 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:06.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.509+0000 7f57d6ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f57d00b00b0 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:06.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.509+0000 7f57d6ec9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00b0500 con 0x7f57d006ab80 2024-03-25T18:43:06.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.509+0000 7f57d6ec9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d013ffa0 con 0x7f57d006af60 2024-03-25T18:43:06.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.509+0000 7f57d6ec9640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d00af520 con 0x7f57d00b00b0 2024-03-25T18:43:06.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.509+0000 7f57d6ec9640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:06.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.509+0000 7f57d6ec9640 10 monclient(hunting): authenticate will time out at 6292.697739s 2024-03-25T18:43:06.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.520+0000 7f57d4c65640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f57d006ab80 0x7f57d00ad880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41214/0 (socket says 172.21.15.32:41214) 2024-03-25T18:43:06.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.520+0000 7f57d4c65640 1 -- 172.21.15.32:0/4102284147 learned_addr learned my addr 172.21.15.32:0/4102284147 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:06.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.533+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2480465562 0 0) 0x7f57d013ffa0 con 0x7f57d006af60 2024-03-25T18:43:06.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.533+0000 7f57cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:43:06.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.533+0000 7f57cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:06.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.533+0000 7f57cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:06.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.533+0000 7f57cf7fe640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:43:06.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.533+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57c0003520 con 0x7f57d006af60 2024-03-25T18:43:06.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.538+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2484487089 0 0) 0x7f57d00b0500 con 0x7f57d006ab80 2024-03-25T18:43:06.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.538+0000 7f57cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:43:06.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.538+0000 7f57cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:06.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.538+0000 7f57cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:06.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.538+0000 7f57cf7fe640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:43:06.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.538+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d013ffa0 con 0x7f57d006ab80 2024-03-25T18:43:06.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.549+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3144814057 0 0) 0x7f57d00af520 con 0x7f57d00b00b0 2024-03-25T18:43:06.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.549+0000 7f57cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:43:06.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.549+0000 7f57cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:06.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.549+0000 7f57cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:06.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.549+0000 7f57cf7fe640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:43:06.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.549+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d00b0500 con 0x7f57d00b00b0 2024-03-25T18:43:06.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1849903644 0 0) 0x7f57c0003520 con 0x7f57d006af60 2024-03-25T18:43:06.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 >> v1:172.21.15.32:6790/0 conn(0x7f57d00b00b0 legacy=0x7f57d014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:06.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 >> v1:172.21.15.32:6789/0 conn(0x7f57d006ab80 legacy=0x7f57d00ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:06.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:43:06.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 1 monclient: found mon.noname-c 2024-03-25T18:43:06.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:06.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:43:06.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d0072890 con 0x7f57d006af60 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f57bc001690 con 0x7f57d006af60 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57cf7fe640 10 monclient: dump: 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:06.553 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:06.554 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:06.554 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:06.554 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:06.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57d6ec9640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:43:06.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.552+0000 7f57d6ec9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:43:06.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.565+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57bc0017f0 con 0x7f57d006af60 2024-03-25T18:43:06.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.565+0000 7f57cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:06.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.565+0000 7f57cf7fe640 1 -- 172.21.15.32:0/4102284147 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f57bc001c00 con 0x7f57d006af60 2024-03-25T18:43:06.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.565+0000 7f57cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:06.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.565+0000 7f57cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:06.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.565+0000 7f57cf7fe640 10 monclient: dump: 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.565+0000 7f57d6ec9640 10 monclient: get_monmap_and_config success 2024-03-25T18:43:06.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.565+0000 7f57d6ec9640 4 set_mon_vals no callback set 2024-03-25T18:43:06.568 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.566+0000 7f57d6ec9640 10 monclient: shutdown 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.566+0000 7f57d6ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.566+0000 7f57d6ec9640 1 -- 172.21.15.32:0/4102284147 >> v1:172.21.15.136:6789/0 conn(0x7f57d006af60 legacy=0x7f57d013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.566+0000 7f57d6ec9640 1 -- 172.21.15.32:0/4102284147 shutdown_connections 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.566+0000 7f57d6ec9640 1 -- 172.21.15.32:0/4102284147 >> 172.21.15.32:0/4102284147 conn(0x7f57d00b4a80 msgr2=0x7f57d00b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.566+0000 7f57d6ec9640 1 -- 172.21.15.32:0/4102284147 shutdown_connections 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.566+0000 7f57d6ec9640 1 -- 172.21.15.32:0/4102284147 wait complete. 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 10 monclient: build_initial_monmap 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 10 monclient: monmap: 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:06.569 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 1 librados: starting msgr at 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 1 librados: starting objecter 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 1 Processor -- start 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 1 -- start start 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 1 librados: setting wanted keys 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 1 librados: calling monclient init 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.567+0000 7f57d6ec9640 10 monclient: init 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 10 monclient(hunting): picked mon.noname-c con 0x7f57d006ab80 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:06.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 10 monclient(hunting): picked mon.noname-a con 0x7f57d00b00b0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:06.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 10 monclient(hunting): picked mon.noname-b con 0x7f57d01d7470 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:06.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d01457b0 con 0x7f57d00b00b0 2024-03-25T18:43:06.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d01d50b0 con 0x7f57d01d7470 2024-03-25T18:43:06.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57d01467b0 con 0x7f57d006ab80 2024-03-25T18:43:06.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:06.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.568+0000 7f57d6ec9640 10 monclient(hunting): authenticate will time out at 6292.757336s 2024-03-25T18:43:06.586 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.586+0000 7f57cffff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f57d00b00b0 0x7f57d01d49a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:41268/0 (socket says 172.21.15.32:41268) 2024-03-25T18:43:06.586 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.586+0000 7f57cffff640 1 -- 172.21.15.32:0/682585216 learned_addr learned my addr 172.21.15.32:0/682585216 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:06.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.587+0000 7f57cd7fa640 10 client.?.objecter ms_handle_connect 0x7f57d01d7470 2024-03-25T18:43:06.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.587+0000 7f57cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:06.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.589+0000 7f57cd7fa640 10 client.?.objecter ms_handle_connect 0x7f57d006ab80 2024-03-25T18:43:06.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.589+0000 7f57cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:06.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.590+0000 7f57cd7fa640 10 client.?.objecter ms_handle_connect 0x7f57d00b00b0 2024-03-25T18:43:06.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.590+0000 7f57cd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:06.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.594+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2283277566 0 0) 0x7f57d01457b0 con 0x7f57d00b00b0 2024-03-25T18:43:06.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.594+0000 7f57cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:43:06.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.594+0000 7f57cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:06.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.594+0000 7f57cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:06.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.594+0000 7f57cd7fa640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:43:06.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.594+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57b4004eb0 con 0x7f57d00b00b0 2024-03-25T18:43:06.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.597+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2657022253 0 0) 0x7f57d01467b0 con 0x7f57d006ab80 2024-03-25T18:43:06.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.597+0000 7f57cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:43:06.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.597+0000 7f57cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:06.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.597+0000 7f57cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:06.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.597+0000 7f57cd7fa640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:43:06.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.597+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d01457b0 con 0x7f57d006ab80 2024-03-25T18:43:06.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.599+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009472312 0 0) 0x7f57d01d50b0 con 0x7f57d01d7470 2024-03-25T18:43:06.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.599+0000 7f57cd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:43:06.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.599+0000 7f57cd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:06.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.599+0000 7f57cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:06.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.599+0000 7f57cd7fa640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:43:06.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.600+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57d01467b0 con 0x7f57d01d7470 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.609+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2145430015 0 0) 0x7f57d01457b0 con 0x7f57d006ab80 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.610+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57d01d50b0 con 0x7f57d006ab80 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.610+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f57c4001970 con 0x7f57d006ab80 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.610+0000 7f57cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.610+0000 7f57cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.610+0000 7f57cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.611 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:06.612 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:06.612 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:06.612 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:06.612 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:06.612 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:06.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.610+0000 7f57cd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:43:06.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3601572668 0 0) 0x7f57b4004eb0 con 0x7f57d00b00b0 2024-03-25T18:43:06.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57d01457b0 con 0x7f57d00b00b0 2024-03-25T18:43:06.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f57bc001bc0 con 0x7f57d00b00b0 2024-03-25T18:43:06.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:06.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:43:06.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:43:06.624 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:06.624 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 515577632 0 0) 0x7f57d01467b0 con 0x7f57d01d7470 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f57b4004eb0 con 0x7f57d01d7470 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f57c8001bd0 con 0x7f57d01d7470 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:06.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.623+0000 7f57cd7fa640 10 monclient(hunting): dump: 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:06.626 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:06.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4180059085 0 0) 0x7f57b4004eb0 con 0x7f57d01d7470 2024-03-25T18:43:06.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 >> v1:172.21.15.32:6790/0 conn(0x7f57d006ab80 legacy=0x7f57d0143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:06.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 >> v1:172.21.15.32:6789/0 conn(0x7f57d00b00b0 legacy=0x7f57d01d49a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:06.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:43:06.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 1 monclient: found mon.b 2024-03-25T18:43:06.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:06.631 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57d0145a00 con 0x7f57d01d7470 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 10 monclient: _renew_subs 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d0146290 con 0x7f57d01d7470 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 10 client.4384.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 10 monclient: _renew_subs 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d0143090 con 0x7f57d01d7470 2024-03-25T18:43:06.632 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.630+0000 7f57d6ec9640 1 librados: init done 2024-03-25T18:43:06.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.631+0000 7f57d6ec9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:06.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.631+0000 7f57d6ec9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:06.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.631+0000 7f57d6ec9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:06.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.631+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d00627b0 con 0x7f57d01d7470 2024-03-25T18:43:06.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.927+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f57c8001fe0 con 0x7f57d01d7470 2024-03-25T18:43:06.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.927+0000 7f57cd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.927+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f57c8003b30 con 0x7f57d01d7470 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.927+0000 7f57cd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.927+0000 7f57cd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.927+0000 7f57cd7fa640 10 monclient: dump: 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:06.928 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:06.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.927+0000 7f57ceffd640 4 set_mon_vals no callback set 2024-03-25T18:43:06.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.928+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f57c8019b00 con 0x7f57d01d7470 2024-03-25T18:43:06.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.929+0000 7f57d4c65640 10 monclient: get_auth_request con 0x7f57b406e110 auth_method 0 2024-03-25T18:43:06.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.929+0000 7f57cd7fa640 10 client.4384.objecter ms_handle_connect 0x7f57b406e110 2024-03-25T18:43:06.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.933+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f57c80537e0 con 0x7f57d01d7470 2024-03-25T18:43:06.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.933+0000 7f57cd7fa640 10 client.4384.objecter ms_dispatch 0x7f57d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:43:06.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.933+0000 7f57cd7fa640 3 client.4384.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:43:06.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.933+0000 7f57cd7fa640 3 client.4384.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:43:06.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.933+0000 7f57cd7fa640 20 client.4384.objecter dump_active .. 0 homeless 2024-03-25T18:43:06.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.933+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f57c804f2f0 con 0x7f57d01d7470 2024-03-25T18:43:06.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.933+0000 7f57cd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:06.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:06.933+0000 7f57cd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:43:07.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.185+0000 7f57d6ec9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:07.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.185+0000 7f57d6ec9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:07.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.185+0000 7f57d6ec9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:07.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.185+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f57d0146420 con 0x7f57d01d7470 2024-03-25T18:43:07.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.218+0000 7f57cd7fa640 1 -- 172.21.15.32:0/682585216 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f57c8051220 con 0x7f57d01d7470 2024-03-25T18:43:07.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.218+0000 7f57cd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:07.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.218+0000 7f57cd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:43:07.219 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:43:07.219 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:43:07.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 10 librados: watch_flush enter 2024-03-25T18:43:07.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 10 librados: watch_flush exit 2024-03-25T18:43:07.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 20 client.4384.objecter shutdown clearing up homeless session... 2024-03-25T18:43:07.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 10 client.4384.objecter successfully canceled tick 2024-03-25T18:43:07.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 >> v1:172.21.15.136:6800/1141502808 conn(0x7f57b406e110 legacy=0x7f57b40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:07.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 10 monclient: shutdown 2024-03-25T18:43:07.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:07.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 >> v1:172.21.15.136:6789/0 conn(0x7f57d01d7470 legacy=0x7f57d01d7920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:07.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 shutdown_connections 2024-03-25T18:43:07.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 >> 172.21.15.32:0/682585216 conn(0x7f57d00b4a80 msgr2=0x7f57d006f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:07.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 shutdown_connections 2024-03-25T18:43:07.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 1 -- 172.21.15.32:0/682585216 wait complete. 2024-03-25T18:43:07.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:07.221+0000 7f57d6ec9640 1 librados: shutdown 2024-03-25T18:43:07.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:43:14.040 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:43:14.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.159+0000 7f8756f30640 10 monclient: get_monmap_and_config 2024-03-25T18:43:14.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.159+0000 7f8756f30640 10 monclient: build_initial_monmap 2024-03-25T18:43:14.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.159+0000 7f8756f30640 10 monclient: monmap: 2024-03-25T18:43:14.159 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:14.159 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:14.160 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:43:14.159849+0000 2024-03-25T18:43:14.160 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:43:14.159849+0000 2024-03-25T18:43:14.160 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:14.160 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:14.160 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:14.160 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:43:14.160 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:43:14.160 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:14.161 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.160+0000 7f8756f30640 1 Processor -- start 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.160+0000 7f8756f30640 1 -- start start 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.160+0000 7f8756f30640 10 monclient: init 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.160+0000 7f8756f30640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.160+0000 7f8756f30640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.160+0000 7f8756f30640 10 monclient(hunting): picked mon.noname-c con 0x7f87500a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.161+0000 7f8756f30640 10 monclient(hunting): picked mon.noname-b con 0x7f875013f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.161+0000 7f8756f30640 10 monclient(hunting): picked mon.noname-a con 0x7f87501430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.161+0000 7f8756f30640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8750142520 con 0x7f87501430c0 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.161+0000 7f8756f30640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87500b28a0 con 0x7f87500a98f0 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.161+0000 7f8756f30640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87500b2a30 con 0x7f875013f710 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.161+0000 7f8756f30640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:14.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.161+0000 7f8756f30640 10 monclient(hunting): authenticate will time out at 6300.349708s 2024-03-25T18:43:14.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.166+0000 7f8754ccc640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f87500a98f0 0x7f87500a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:58910/0 (socket says 172.21.15.32:58910) 2024-03-25T18:43:14.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.166+0000 7f8754ccc640 1 -- 172.21.15.32:0/2012553511 learned_addr learned my addr 172.21.15.32:0/2012553511 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:14.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.175+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 699432087 0 0) 0x7f87500b28a0 con 0x7f87500a98f0 2024-03-25T18:43:14.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.175+0000 7f874f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:43:14.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.175+0000 7f874f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:14.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.175+0000 7f874f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:14.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.175+0000 7f874f7fe640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:43:14.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.175+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8740003520 con 0x7f87500a98f0 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1827193864 0 0) 0x7f8740003520 con 0x7f87500a98f0 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 >> v1:172.21.15.32:6790/0 conn(0x7f875013f710 legacy=0x7f87501429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 >> v1:172.21.15.32:6789/0 conn(0x7f87501430c0 legacy=0x7f875014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87500b44c0 con 0x7f87500a98f0 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8744001690 con 0x7f87500a98f0 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:14.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f874f7fe640 10 monclient: dump: 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:14.184 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:14.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f8756f30640 5 monclient: authenticate success, global_id 4387 2024-03-25T18:43:14.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.182+0000 7f8756f30640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:43:14.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.197+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8744001860 con 0x7f87500a98f0 2024-03-25T18:43:14.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.197+0000 7f874f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:14.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.197+0000 7f874f7fe640 1 -- 172.21.15.32:0/2012553511 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8744001cb0 con 0x7f87500a98f0 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.197+0000 7f8756f30640 10 monclient: get_monmap_and_config success 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.197+0000 7f8756f30640 4 set_mon_vals no callback set 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.197+0000 7f874f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.197+0000 7f874f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.197+0000 7f874f7fe640 10 monclient: dump: 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:14.198 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:14.199 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:14.199 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:14.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.198+0000 7f8756f30640 10 monclient: shutdown 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.198+0000 7f8756f30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.198+0000 7f8756f30640 1 -- 172.21.15.32:0/2012553511 >> v1:172.21.15.136:6789/0 conn(0x7f87500a98f0 legacy=0x7f87500a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.198+0000 7f8756f30640 1 -- 172.21.15.32:0/2012553511 shutdown_connections 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.198+0000 7f8756f30640 1 -- 172.21.15.32:0/2012553511 >> 172.21.15.32:0/2012553511 conn(0x7f87500b7ea0 msgr2=0x7f87500b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.198+0000 7f8756f30640 1 -- 172.21.15.32:0/2012553511 shutdown_connections 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.198+0000 7f8756f30640 1 -- 172.21.15.32:0/2012553511 wait complete. 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.199+0000 7f8756f30640 10 monclient: build_initial_monmap 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.199+0000 7f8756f30640 10 monclient: monmap: 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:43:14.200 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:14.201 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:43:14.201 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:14.201 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:43:14.201 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:43:14.201 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:14.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.200+0000 7f8756f30640 1 librados: starting msgr at 2024-03-25T18:43:14.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.200+0000 7f8756f30640 1 librados: starting objecter 2024-03-25T18:43:14.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.200+0000 7f8756f30640 1 Processor -- start 2024-03-25T18:43:14.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.200+0000 7f8756f30640 1 -- start start 2024-03-25T18:43:14.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.200+0000 7f8756f30640 1 librados: setting wanted keys 2024-03-25T18:43:14.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.200+0000 7f8756f30640 1 librados: calling monclient init 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.200+0000 7f8756f30640 10 monclient: init 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 10 monclient(hunting): picked mon.noname-b con 0x7f87500a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 10 monclient(hunting): picked mon.noname-c con 0x7f875013f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 10 monclient(hunting): picked mon.noname-a con 0x7f87501430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87501df1d0 con 0x7f87501430c0 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f87500708d0 con 0x7f87500a98f0 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8750070cf0 con 0x7f875013f710 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:14.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.201+0000 7f8756f30640 10 monclient(hunting): authenticate will time out at 6300.390284s 2024-03-25T18:43:14.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.210+0000 7f874ffff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f875013f710 0x7f875006ebc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:53366/0 (socket says 172.21.15.32:53366) 2024-03-25T18:43:14.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.210+0000 7f874ffff640 1 -- 172.21.15.32:0/3966583514 learned_addr learned my addr 172.21.15.32:0/3966583514 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:14.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.212+0000 7f874d7fa640 10 client.?.objecter ms_handle_connect 0x7f875013f710 2024-03-25T18:43:14.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.213+0000 7f874d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:14.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.217+0000 7f874d7fa640 10 client.?.objecter ms_handle_connect 0x7f87500a98f0 2024-03-25T18:43:14.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.217+0000 7f874d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:14.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.224+0000 7f874d7fa640 10 client.?.objecter ms_handle_connect 0x7f87501430c0 2024-03-25T18:43:14.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.224+0000 7f874d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:14.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.225+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 15188096 0 0) 0x7f8750070cf0 con 0x7f875013f710 2024-03-25T18:43:14.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.225+0000 7f874d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:43:14.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.225+0000 7f874d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:14.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.225+0000 7f874d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:14.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.225+0000 7f874d7fa640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:43:14.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.225+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8734004eb0 con 0x7f875013f710 2024-03-25T18:43:14.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.237+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 636014942 0 0) 0x7f87501df1d0 con 0x7f87501430c0 2024-03-25T18:43:14.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.237+0000 7f874d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:43:14.238 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.237+0000 7f874d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:14.238 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.237+0000 7f874d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:14.238 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.237+0000 7f874d7fa640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:43:14.238 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.237+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8750070cf0 con 0x7f87501430c0 2024-03-25T18:43:14.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.296+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593839948 0 0) 0x7f87500708d0 con 0x7f87500a98f0 2024-03-25T18:43:14.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.296+0000 7f874d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:43:14.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.296+0000 7f874d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:14.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.296+0000 7f874d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:14.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.296+0000 7f874d7fa640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:43:14.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.296+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87501df1d0 con 0x7f87500a98f0 2024-03-25T18:43:14.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.304+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4256222093 0 0) 0x7f8750070cf0 con 0x7f87501430c0 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.304+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87500708d0 con 0x7f87501430c0 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.304+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8748001b10 con 0x7f87501430c0 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.304+0000 7f874d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.304+0000 7f874d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.304+0000 7f874d7fa640 10 monclient(hunting): dump: 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:14.305 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:14.306 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:14.306 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.304+0000 7f874d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:43:14.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.313+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4014708138 0 0) 0x7f87501df1d0 con 0x7f87500a98f0 2024-03-25T18:43:14.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.313+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8750070cf0 con 0x7f87500a98f0 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.313+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f87440019d0 con 0x7f87500a98f0 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.313+0000 7f874d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.313+0000 7f874d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.313+0000 7f874d7fa640 10 monclient(hunting): dump: 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:14.314 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:14.315 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3731853318 0 0) 0x7f87500708d0 con 0x7f87501430c0 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 >> v1:172.21.15.32:6790/0 conn(0x7f875013f710 legacy=0x7f875006ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 >> v1:172.21.15.136:6789/0 conn(0x7f87500a98f0 legacy=0x7f87501df6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 1 monclient: found mon.a 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8750070e80 con 0x7f87501430c0 2024-03-25T18:43:14.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f874d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 5 monclient: authenticate success, global_id 4407 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 10 monclient: _renew_subs 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87501dfdc0 con 0x7f87501430c0 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 10 monclient: _renew_subs 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f87501e78c0 con 0x7f87501430c0 2024-03-25T18:43:14.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.323+0000 7f8756f30640 1 librados: init done 2024-03-25T18:43:14.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.324+0000 7f8756f30640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:14.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.324+0000 7f8756f30640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:14.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.324+0000 7f8756f30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:14.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.324+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87500627b0 con 0x7f87501430c0 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.637+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8748002000 con 0x7f87501430c0 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.637+0000 7f874d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.637+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8748003b10 con 0x7f87501430c0 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.637+0000 7f874d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.637+0000 7f874d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.637+0000 7f874d7fa640 10 monclient: dump: 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:14.638 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:14.639 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:14.639 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:14.639 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:14.639 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:14.639 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:14.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.637+0000 7f874effd640 4 set_mon_vals no callback set 2024-03-25T18:43:14.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.638+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f8748003d30 con 0x7f87501430c0 2024-03-25T18:43:14.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.638+0000 7f8754ccc640 10 monclient: get_auth_request con 0x7f873406df80 auth_method 0 2024-03-25T18:43:14.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.639+0000 7f874d7fa640 10 client.4407.objecter ms_handle_connect 0x7f873406df80 2024-03-25T18:43:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.644+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f8748053800 con 0x7f87501430c0 2024-03-25T18:43:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.644+0000 7f874d7fa640 10 client.4407.objecter ms_dispatch 0x7f8750000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:43:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.644+0000 7f874d7fa640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:43:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.644+0000 7f874d7fa640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:43:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.645+0000 7f874d7fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-25T18:43:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.645+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f874804f310 con 0x7f87501430c0 2024-03-25T18:43:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.645+0000 7f874d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.645+0000 7f874d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:43:14.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.908+0000 7f8756f30640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:14.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.908+0000 7f8756f30640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:14.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.908+0000 7f8756f30640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:14.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.908+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f875013e100 con 0x7f87501430c0 2024-03-25T18:43:14.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.945+0000 7f874d7fa640 1 -- 172.21.15.32:0/3966583514 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f8748051240 con 0x7f87501430c0 2024-03-25T18:43:14.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.945+0000 7f874d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:14.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.945+0000 7f874d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:43:14.946 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:43:14.946 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:43:14.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.947+0000 7f8756f30640 10 librados: watch_flush enter 2024-03-25T18:43:14.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 10 librados: watch_flush exit 2024-03-25T18:43:14.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 10 client.4407.objecter successfully canceled tick 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 >> v1:172.21.15.136:6800/1141502808 conn(0x7f873406df80 legacy=0x7f8734070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 10 monclient: shutdown 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 >> v1:172.21.15.32:6789/0 conn(0x7f87501430c0 legacy=0x7f875006f2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 shutdown_connections 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 >> 172.21.15.32:0/3966583514 conn(0x7f87500b7ea0 msgr2=0x7f8750074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 shutdown_connections 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 1 -- 172.21.15.32:0/3966583514 wait complete. 2024-03-25T18:43:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:14.948+0000 7f8756f30640 1 librados: shutdown 2024-03-25T18:43:14.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:43:21.766 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:43:21.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.886+0000 7f751f4e6640 10 monclient: get_monmap_and_config 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.886+0000 7f751f4e6640 10 monclient: build_initial_monmap 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.886+0000 7f751f4e6640 10 monclient: monmap: 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:43:21.886732+0000 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:43:21.886732+0000 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:21.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.887+0000 7f751f4e6640 1 Processor -- start 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.887+0000 7f751f4e6640 1 -- start start 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.887+0000 7f751f4e6640 10 monclient: init 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.887+0000 7f751f4e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.887+0000 7f751f4e6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.887+0000 7f751f4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f75180a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.888+0000 7f751f4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f751813f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.888+0000 7f751f4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f75181430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.888+0000 7f751f4e6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7518142520 con 0x7f75180a98f0 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.888+0000 7f751f4e6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180b28a0 con 0x7f75181430c0 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.888+0000 7f751f4e6640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180b2a30 con 0x7f751813f710 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.888+0000 7f751f4e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:21.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.888+0000 7f751f4e6640 10 monclient(hunting): authenticate will time out at 6308.076716s 2024-03-25T18:43:21.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.901+0000 7f751ca81640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f751813f710 0x7f75181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:56924/0 (socket says 172.21.15.32:56924) 2024-03-25T18:43:21.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.901+0000 7f751ca81640 1 -- 172.21.15.32:0/1506825246 learned_addr learned my addr 172.21.15.32:0/1506825246 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:21.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.921+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1283124280 0 0) 0x7f75180b28a0 con 0x7f75181430c0 2024-03-25T18:43:21.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.921+0000 7f750ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:43:21.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.921+0000 7f750ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:21.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.921+0000 7f750ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:21.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.921+0000 7f750ffff640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:43:21.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.921+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7504003520 con 0x7f75181430c0 2024-03-25T18:43:21.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4277026174 0 0) 0x7f7504003520 con 0x7f75181430c0 2024-03-25T18:43:21.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 >> v1:172.21.15.32:6790/0 conn(0x7f751813f710 legacy=0x7f75181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:21.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 >> v1:172.21.15.32:6789/0 conn(0x7f75180a98f0 legacy=0x7f75180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:21.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:43:21.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 1 monclient: found mon.noname-c 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180b44c0 con 0x7f75181430c0 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 10 monclient: _finish_auth 0 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7510001a10 con 0x7f75181430c0 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f751f4e6640 5 monclient: authenticate success, global_id 4393 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f751f4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f750ffff640 10 monclient: dump: 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:21.932 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:21.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.931+0000 7f751f4e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.955+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7510001b70 con 0x7f75181430c0 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.955+0000 7f750ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.955+0000 7f750ffff640 1 -- 172.21.15.32:0/1506825246 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7510001fc0 con 0x7f75181430c0 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.955+0000 7f750ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.955+0000 7f750ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.955+0000 7f750ffff640 10 monclient: dump: 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:21.956 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:21.957 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:21.957 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:21.957 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:21.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.955+0000 7f751f4e6640 10 monclient: get_monmap_and_config success 2024-03-25T18:43:21.957 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.955+0000 7f751f4e6640 4 set_mon_vals no callback set 2024-03-25T18:43:21.959 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.956+0000 7f751f4e6640 10 monclient: shutdown 2024-03-25T18:43:21.959 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.956+0000 7f751f4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:21.959 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.956+0000 7f751f4e6640 1 -- 172.21.15.32:0/1506825246 >> v1:172.21.15.136:6789/0 conn(0x7f75181430c0 legacy=0x7f751814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:21.959 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.956+0000 7f751f4e6640 1 -- 172.21.15.32:0/1506825246 shutdown_connections 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.956+0000 7f751f4e6640 1 -- 172.21.15.32:0/1506825246 >> 172.21.15.32:0/1506825246 conn(0x7f75180b7ea0 msgr2=0x7f75180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.956+0000 7f751f4e6640 1 -- 172.21.15.32:0/1506825246 shutdown_connections 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.956+0000 7f751f4e6640 1 -- 172.21.15.32:0/1506825246 wait complete. 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.957+0000 7f751f4e6640 10 monclient: build_initial_monmap 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.957+0000 7f751f4e6640 10 monclient: monmap: 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:43:21.960 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 1 librados: starting msgr at 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 1 librados: starting objecter 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 1 Processor -- start 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 1 -- start start 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 1 librados: setting wanted keys 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 1 librados: calling monclient init 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 10 monclient: init 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:43:21.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 10 monclient(hunting): picked mon.noname-c con 0x7f75180a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:21.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.958+0000 7f751f4e6640 10 monclient(hunting): picked mon.noname-a con 0x7f751813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:21.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.959+0000 7f751f4e6640 10 monclient(hunting): picked mon.noname-b con 0x7f75181430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:21.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.959+0000 7f751f4e6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7518136ad0 con 0x7f751813f710 2024-03-25T18:43:21.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.959+0000 7f751f4e6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180b7520 con 0x7f75181430c0 2024-03-25T18:43:21.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.959+0000 7f751f4e6640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7518135810 con 0x7f75180a98f0 2024-03-25T18:43:21.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.959+0000 7f751f4e6640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:21.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.959+0000 7f751f4e6640 10 monclient(hunting): authenticate will time out at 6308.147778s 2024-03-25T18:43:21.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.970+0000 7f751d282640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f75180a98f0 0x7f7518135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:56968/0 (socket says 172.21.15.32:56968) 2024-03-25T18:43:21.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.970+0000 7f751d282640 1 -- 172.21.15.32:0/490966823 learned_addr learned my addr 172.21.15.32:0/490966823 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:21.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.972+0000 7f750dffb640 10 client.?.objecter ms_handle_connect 0x7f75180a98f0 2024-03-25T18:43:21.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.972+0000 7f750dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:21.979 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.979+0000 7f750dffb640 10 client.?.objecter ms_handle_connect 0x7f75181430c0 2024-03-25T18:43:21.979 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.979+0000 7f750dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:21.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.986+0000 7f750dffb640 10 client.?.objecter ms_handle_connect 0x7f751813f710 2024-03-25T18:43:21.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.986+0000 7f750dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:21.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.990+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1601752226 0 0) 0x7f7518135810 con 0x7f75180a98f0 2024-03-25T18:43:21.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.990+0000 7f750dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:43:21.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.990+0000 7f750dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:21.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.990+0000 7f750dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:21.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.990+0000 7f750dffb640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:43:21.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.990+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74fc004eb0 con 0x7f75180a98f0 2024-03-25T18:43:21.996 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.995+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421242445 0 0) 0x7f7518136ad0 con 0x7f751813f710 2024-03-25T18:43:21.996 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.995+0000 7f750dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:43:21.996 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.995+0000 7f750dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:21.996 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.995+0000 7f750dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:21.996 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.995+0000 7f750dffb640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:43:21.996 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:21.996+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74fc0051d0 con 0x7f751813f710 2024-03-25T18:43:22.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.000+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616461994 0 0) 0x7f75180b7520 con 0x7f75181430c0 2024-03-25T18:43:22.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.000+0000 7f750dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:43:22.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.000+0000 7f750dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:22.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.000+0000 7f750dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:22.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.000+0000 7f750dffb640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:43:22.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.000+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7518136ad0 con 0x7f75181430c0 2024-03-25T18:43:22.003 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.003+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1006251965 0 0) 0x7f74fc004eb0 con 0x7f75180a98f0 2024-03-25T18:43:22.003 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.003+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75180b7520 con 0x7f75180a98f0 2024-03-25T18:43:22.003 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.003+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f75140019f0 con 0x7f75180a98f0 2024-03-25T18:43:22.003 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.003+0000 7f750dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:22.003 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.003+0000 7f750dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:43:22.003 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.003+0000 7f750dffb640 10 monclient(hunting): dump: 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:22.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.003+0000 7f750dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.004+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1682849883 0 0) 0x7f74fc0051d0 con 0x7f751813f710 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.004+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74fc004eb0 con 0x7f751813f710 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.005+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f75080018a0 con 0x7f751813f710 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.005+0000 7f750dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.005+0000 7f750dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.005+0000 7f750dffb640 10 monclient(hunting): dump: 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:22.005 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:22.006 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:22.006 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:22.006 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:22.006 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:22.006 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:22.006 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:22.006 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:22.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1300303078 0 0) 0x7f74fc004eb0 con 0x7f751813f710 2024-03-25T18:43:22.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 >> v1:172.21.15.32:6790/0 conn(0x7f75180a98f0 legacy=0x7f7518135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:22.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 >> v1:172.21.15.136:6789/0 conn(0x7f75181430c0 legacy=0x7f7518130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:22.019 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 1 monclient: found mon.a 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180b6180 con 0x7f751813f710 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 10 monclient: _finish_auth 0 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f750dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f751f4e6640 5 monclient: authenticate success, global_id 4413 2024-03-25T18:43:22.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f751f4e6640 10 monclient: _renew_subs 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f751f4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75180b7a40 con 0x7f751813f710 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f751f4e6640 10 client.4413.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f751f4e6640 10 monclient: _renew_subs 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.017+0000 7f751f4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.018+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7518131840 con 0x7f751813f710 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.018+0000 7f751f4e6640 1 librados: init done 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.018+0000 7f751f4e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.018+0000 7f751f4e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:22.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.019+0000 7f751f4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:22.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.019+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7518136f60 con 0x7f751813f710 2024-03-25T18:43:22.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.371+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7508001cb0 con 0x7f751813f710 2024-03-25T18:43:22.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.371+0000 7f750dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:22.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.371+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f75080047e0 con 0x7f751813f710 2024-03-25T18:43:22.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.371+0000 7f750dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:22.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.371+0000 7f750dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:43:22.372 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.371+0000 7f750dffb640 10 monclient: dump: 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.371+0000 7f750f7fe640 4 set_mon_vals no callback set 2024-03-25T18:43:22.373 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.372+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f75080049c0 con 0x7f751813f710 2024-03-25T18:43:22.374 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.373+0000 7f751d282640 10 monclient: get_auth_request con 0x7f74fc06e130 auth_method 0 2024-03-25T18:43:22.374 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.373+0000 7f750dffb640 10 client.4413.objecter ms_handle_connect 0x7f74fc06e130 2024-03-25T18:43:22.379 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.379+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f7508054550 con 0x7f751813f710 2024-03-25T18:43:22.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.379+0000 7f750dffb640 10 client.4413.objecter ms_dispatch 0x7f7518000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:43:22.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.379+0000 7f750dffb640 3 client.4413.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:43:22.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.379+0000 7f750dffb640 3 client.4413.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:43:22.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.379+0000 7f750dffb640 20 client.4413.objecter dump_active .. 0 homeless 2024-03-25T18:43:22.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.379+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f750804fe90 con 0x7f751813f710 2024-03-25T18:43:22.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.379+0000 7f750dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:22.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.379+0000 7f750dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:43:22.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.615+0000 7f751f4e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:22.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.615+0000 7f751f4e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:22.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.615+0000 7f751f4e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:43:22.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.615+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f75180b6310 con 0x7f751813f710 2024-03-25T18:43:22.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.683+0000 7f750dffb640 1 -- 172.21.15.32:0/490966823 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f7508051f90 con 0x7f751813f710 2024-03-25T18:43:22.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.683+0000 7f750dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:22.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.683+0000 7f750dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:43:22.683 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:43:22.683 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 10 librados: watch_flush enter 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 10 librados: watch_flush exit 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 20 client.4413.objecter shutdown clearing up homeless session... 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 10 client.4413.objecter successfully canceled tick 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 >> v1:172.21.15.136:6800/1141502808 conn(0x7f74fc06e130 legacy=0x7f74fc0705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 10 monclient: shutdown 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 >> v1:172.21.15.32:6789/0 conn(0x7f751813f710 legacy=0x7f75180b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 shutdown_connections 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 >> 172.21.15.32:0/490966823 conn(0x7f75180b7ea0 msgr2=0x7f7518074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 shutdown_connections 2024-03-25T18:43:22.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 1 -- 172.21.15.32:0/490966823 wait complete. 2024-03-25T18:43:22.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:22.685+0000 7f751f4e6640 1 librados: shutdown 2024-03-25T18:43:22.696 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:43:27.684 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:04:40.011 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:27.691 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:04:40.019 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:27.732 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:04:40.060 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:27.741 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:04:40.058 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:27.745 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:04:40.062 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:27.747 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:04:40.075 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:27.752 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:04:40.080 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:27.777 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:04:40.094 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:27.786 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:04:40.114 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:29.503 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:43:29.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.620+0000 7f14c8c0e640 10 monclient: get_monmap_and_config 2024-03-25T18:43:29.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.620+0000 7f14c8c0e640 10 monclient: build_initial_monmap 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.620+0000 7f14c8c0e640 10 monclient: monmap: 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:43:29.621541+0000 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:43:29.621541+0000 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.620+0000 7f14c8c0e640 1 Processor -- start 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.620+0000 7f14c8c0e640 1 -- start start 2024-03-25T18:43:29.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.620+0000 7f14c8c0e640 10 monclient: init 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 10 monclient(hunting): picked mon.noname-b con 0x7f14c406c380 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 10 monclient(hunting): picked mon.noname-a con 0x7f14c40b12d0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 10 monclient(hunting): picked mon.noname-c con 0x7f14c40aedf0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c40b16f0 con 0x7f14c40b12d0 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c40adfe0 con 0x7f14c40aedf0 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c40add00 con 0x7f14c406c380 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:29.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.621+0000 7f14c8c0e640 10 monclient(hunting): authenticate will time out at 6315.810920s 2024-03-25T18:43:29.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.635+0000 7f14c259c640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f14c406c380 0x7f14c40b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:40998/0 (socket says 172.21.15.32:40998) 2024-03-25T18:43:29.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.635+0000 7f14c259c640 1 -- 172.21.15.32:0/1205830704 learned_addr learned my addr 172.21.15.32:0/1205830704 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:29.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.649+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3722220706 0 0) 0x7f14c40adfe0 con 0x7f14c40aedf0 2024-03-25T18:43:29.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.649+0000 7f14c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:43:29.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.649+0000 7f14c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:29.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.649+0000 7f14c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:29.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.649+0000 7f14c1d9b640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:43:29.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.649+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14b0003520 con 0x7f14c40aedf0 2024-03-25T18:43:29.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.657+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3066117336 0 0) 0x7f14c40b16f0 con 0x7f14c40b12d0 2024-03-25T18:43:29.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.657+0000 7f14c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:43:29.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.657+0000 7f14c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:29.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.657+0000 7f14c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:29.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.657+0000 7f14c1d9b640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:43:29.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.657+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c40adfe0 con 0x7f14c40b12d0 2024-03-25T18:43:29.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.659+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 910550612 0 0) 0x7f14c40add00 con 0x7f14c406c380 2024-03-25T18:43:29.661 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.659+0000 7f14c1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:43:29.661 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.659+0000 7f14c1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:29.661 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.659+0000 7f14c1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:29.661 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.659+0000 7f14c1d9b640 10 monclient(hunting): my global_id is 4403 2024-03-25T18:43:29.661 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.659+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c40b16f0 con 0x7f14c406c380 2024-03-25T18:43:29.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 727060723 0 0) 0x7f14c40adfe0 con 0x7f14c40b12d0 2024-03-25T18:43:29.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 >> v1:172.21.15.32:6790/0 conn(0x7f14c406c380 legacy=0x7f14c40b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:29.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 >> v1:172.21.15.136:6789/0 conn(0x7f14c40aedf0 legacy=0x7f14c414e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:29.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:43:29.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 1 monclient: found mon.noname-a 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c40ae2c0 con 0x7f14c40b12d0 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f14ac001670 con 0x7f14c40b12d0 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c1d9b640 10 monclient: dump: 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:29.672 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:29.673 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:29.673 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:29.673 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:29.673 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:29.673 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:29.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c8c0e640 5 monclient: authenticate success, global_id 4416 2024-03-25T18:43:29.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.670+0000 7f14c8c0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:43:29.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.689+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14ac0017d0 con 0x7f14c40b12d0 2024-03-25T18:43:29.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.689+0000 7f14c1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:29.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.689+0000 7f14c1d9b640 1 -- 172.21.15.32:0/1205830704 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f14ac001b60 con 0x7f14c40b12d0 2024-03-25T18:43:29.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.689+0000 7f14c8c0e640 10 monclient: get_monmap_and_config success 2024-03-25T18:43:29.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.689+0000 7f14c8c0e640 4 set_mon_vals no callback set 2024-03-25T18:43:29.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.689+0000 7f14c1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:29.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.689+0000 7f14c1d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:43:29.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.689+0000 7f14c1d9b640 10 monclient: dump: 2024-03-25T18:43:29.691 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:29.692 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:29.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.690+0000 7f14c8c0e640 10 monclient: shutdown 2024-03-25T18:43:29.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.690+0000 7f14c8c0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.690+0000 7f14c8c0e640 1 -- 172.21.15.32:0/1205830704 >> v1:172.21.15.32:6789/0 conn(0x7f14c40b12d0 legacy=0x7f14c40ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.690+0000 7f14c8c0e640 1 -- 172.21.15.32:0/1205830704 shutdown_connections 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.690+0000 7f14c8c0e640 1 -- 172.21.15.32:0/1205830704 >> 172.21.15.32:0/1205830704 conn(0x7f14c40b5d10 msgr2=0x7f14c40b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.690+0000 7f14c8c0e640 1 -- 172.21.15.32:0/1205830704 shutdown_connections 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.690+0000 7f14c8c0e640 1 -- 172.21.15.32:0/1205830704 wait complete. 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 10 monclient: build_initial_monmap 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 10 monclient: monmap: 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:43:29.696 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 1 librados: starting msgr at 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 1 librados: starting objecter 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 1 Processor -- start 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 1 -- start start 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 1 librados: setting wanted keys 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 1 librados: calling monclient init 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.691+0000 7f14c8c0e640 10 monclient: init 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 10 monclient(hunting): picked mon.noname-a con 0x7f14c406c380 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:29.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 10 monclient(hunting): picked mon.noname-b con 0x7f14c40aedf0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:29.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 10 monclient(hunting): picked mon.noname-c con 0x7f14c40b12d0 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:29.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c406c8e0 con 0x7f14c406c380 2024-03-25T18:43:29.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c41c4cf0 con 0x7f14c40aedf0 2024-03-25T18:43:29.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14c41c0ef0 con 0x7f14c40b12d0 2024-03-25T18:43:29.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:29.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.692+0000 7f14c8c0e640 10 monclient(hunting): authenticate will time out at 6315.882012s 2024-03-25T18:43:29.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.706+0000 7f14b9d9b640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f14c40aedf0 0x7f14c41c1e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:36822/0 (socket says 172.21.15.32:36822) 2024-03-25T18:43:29.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.706+0000 7f14b9d9b640 1 -- 172.21.15.32:0/2308441437 learned_addr learned my addr 172.21.15.32:0/2308441437 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:29.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.707+0000 7f14bb7fe640 10 client.?.objecter ms_handle_connect 0x7f14c40aedf0 2024-03-25T18:43:29.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.707+0000 7f14bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:29.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.711+0000 7f14bb7fe640 10 client.?.objecter ms_handle_connect 0x7f14c40b12d0 2024-03-25T18:43:29.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.711+0000 7f14bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:29.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.721+0000 7f14bb7fe640 10 client.?.objecter ms_handle_connect 0x7f14c406c380 2024-03-25T18:43:29.722 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.721+0000 7f14bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:29.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.723+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091176063 0 0) 0x7f14c41c4cf0 con 0x7f14c40aedf0 2024-03-25T18:43:29.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.723+0000 7f14bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:43:29.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.723+0000 7f14bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:29.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.723+0000 7f14bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:29.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.723+0000 7f14bb7fe640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:43:29.724 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.723+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14a4004eb0 con 0x7f14c40aedf0 2024-03-25T18:43:29.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.734+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2930113663 0 0) 0x7f14c406c8e0 con 0x7f14c406c380 2024-03-25T18:43:29.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.734+0000 7f14bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:43:29.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.734+0000 7f14bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:29.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.734+0000 7f14bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:29.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.734+0000 7f14bb7fe640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:43:29.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.734+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c41c4cf0 con 0x7f14c406c380 2024-03-25T18:43:29.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.736+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167992468 0 0) 0x7f14c41c0ef0 con 0x7f14c40b12d0 2024-03-25T18:43:29.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.736+0000 7f14bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:43:29.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.736+0000 7f14bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:29.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.736+0000 7f14bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:29.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.736+0000 7f14bb7fe640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:43:29.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.736+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f14c406c8e0 con 0x7f14c40b12d0 2024-03-25T18:43:29.744 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.743+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4751596 0 0) 0x7f14a4004eb0 con 0x7f14c40aedf0 2024-03-25T18:43:29.744 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.743+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14c41c0ef0 con 0x7f14c40aedf0 2024-03-25T18:43:29.744 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.743+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f14ac001c80 con 0x7f14c40aedf0 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.743+0000 7f14bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.743+0000 7f14bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.743+0000 7f14bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:29.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.743+0000 7f14bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:43:29.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.751+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2980438738 0 0) 0x7f14c41c4cf0 con 0x7f14c406c380 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.751+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f14a4004eb0 con 0x7f14c406c380 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.751+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f14b4001870 con 0x7f14c406c380 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.751+0000 7f14bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.752+0000 7f14bb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.752+0000 7f14bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:29.753 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:29.754 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:29.754 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:29.754 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3819300444 0 0) 0x7f14c41c0ef0 con 0x7f14c40aedf0 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 >> v1:172.21.15.32:6790/0 conn(0x7f14c40b12d0 legacy=0x7f14c41c2980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 >> v1:172.21.15.32:6789/0 conn(0x7f14c406c380 legacy=0x7f14c41c07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 1 monclient: found mon.b 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c41c4830 con 0x7f14c40aedf0 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:29.756 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 5 monclient: authenticate success, global_id 4402 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 10 monclient: _renew_subs 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c41d7410 con 0x7f14c40aedf0 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 10 client.4402.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 10 monclient: _renew_subs 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c41c2750 con 0x7f14c40aedf0 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.754+0000 7f14c8c0e640 1 librados: init done 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.755+0000 7f14c8c0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.755+0000 7f14c8c0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.755+0000 7f14c8c0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:29.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:29.755+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c40627b0 con 0x7f14c40aedf0 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.037+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f14ac0038a0 con 0x7f14c40aedf0 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.037+0000 7f14bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.038+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f14ac003c70 con 0x7f14c40aedf0 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.038+0000 7f14bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.038+0000 7f14bb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.038+0000 7f14bb7fe640 10 monclient: dump: 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:30.039 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:30.040 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:30.040 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:30.040 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:30.040 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:30.040 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:30.040 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:30.040 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:30.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.038+0000 7f14c159a640 4 set_mon_vals no callback set 2024-03-25T18:43:30.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.038+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f14ac003e90 con 0x7f14c40aedf0 2024-03-25T18:43:30.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.039+0000 7f14c259c640 10 monclient: get_auth_request con 0x7f14a406df60 auth_method 0 2024-03-25T18:43:30.041 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.039+0000 7f14bb7fe640 10 client.4402.objecter ms_handle_connect 0x7f14a406df60 2024-03-25T18:43:30.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.045+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f14ac0548e0 con 0x7f14c40aedf0 2024-03-25T18:43:30.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.045+0000 7f14bb7fe640 10 client.4402.objecter ms_dispatch 0x7f14c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:43:30.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.045+0000 7f14bb7fe640 3 client.4402.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:43:30.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.045+0000 7f14bb7fe640 3 client.4402.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:43:30.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.045+0000 7f14bb7fe640 20 client.4402.objecter dump_active .. 0 homeless 2024-03-25T18:43:30.047 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.046+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f14ac0503f0 con 0x7f14c40aedf0 2024-03-25T18:43:30.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.046+0000 7f14bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:30.048 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.046+0000 7f14bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:43:30.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.275+0000 7f14c8c0e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:30.277 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.275+0000 7f14c8c0e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:30.277 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.275+0000 7f14c8c0e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:30.277 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.275+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f14c41c4ea0 con 0x7f14c40aedf0 2024-03-25T18:43:30.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.295+0000 7f14bb7fe640 1 -- 172.21.15.32:0/2308441437 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f14c41c4ea0 con 0x7f14c40aedf0 2024-03-25T18:43:30.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.295+0000 7f14bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:30.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.295+0000 7f14bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:43:30.297 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:43:30.297 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:43:30.299 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 10 librados: watch_flush enter 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 10 librados: watch_flush exit 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 20 client.4402.objecter shutdown clearing up homeless session... 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 10 client.4402.objecter successfully canceled tick 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 >> v1:172.21.15.136:6800/1141502808 conn(0x7f14a406df60 legacy=0x7f14a4070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 10 monclient: shutdown 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 >> v1:172.21.15.136:6789/0 conn(0x7f14c40aedf0 legacy=0x7f14c41c1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 shutdown_connections 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 >> 172.21.15.32:0/2308441437 conn(0x7f14c40b5d10 msgr2=0x7f14c41cd520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 shutdown_connections 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 1 -- 172.21.15.32:0/2308441437 wait complete. 2024-03-25T18:43:30.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:30.298+0000 7f14c8c0e640 1 librados: shutdown 2024-03-25T18:43:30.313 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:43:37.121 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:43:37.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.230+0000 7fe39b8ae640 10 monclient: get_monmap_and_config 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.230+0000 7fe39b8ae640 10 monclient: build_initial_monmap 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.230+0000 7fe39b8ae640 10 monclient: monmap: 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:43:37.232184+0000 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:43:37.232184+0000 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:37.232 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:43:37.233 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:43:37.233 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:37.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.231+0000 7fe39b8ae640 1 Processor -- start 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.231+0000 7fe39b8ae640 1 -- start start 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.231+0000 7fe39b8ae640 10 monclient: init 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 10 monclient(hunting): picked mon.noname-c con 0x7fe3940a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 10 monclient(hunting): picked mon.noname-b con 0x7fe39413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 10 monclient(hunting): picked mon.noname-a con 0x7fe3941430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe394142520 con 0x7fe3941430c0 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3940b28a0 con 0x7fe3940a98f0 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3940b2a30 con 0x7fe39413f710 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:37.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.232+0000 7fe39b8ae640 10 monclient(hunting): authenticate will time out at 6323.421937s 2024-03-25T18:43:37.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.243+0000 7fe39964a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe3940a98f0 0x7fe3940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:57846/0 (socket says 172.21.15.32:57846) 2024-03-25T18:43:37.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.243+0000 7fe39964a640 1 -- 172.21.15.32:0/297400398 learned_addr learned my addr 172.21.15.32:0/297400398 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:37.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.253+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216774478 0 0) 0x7fe3940b28a0 con 0x7fe3940a98f0 2024-03-25T18:43:37.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.253+0000 7fe38bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:43:37.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.253+0000 7fe38bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:37.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.253+0000 7fe38bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:37.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.253+0000 7fe38bfff640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:43:37.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.253+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe380003520 con 0x7fe3940a98f0 2024-03-25T18:43:37.277 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.277+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 539110485 0 0) 0x7fe3940b2a30 con 0x7fe39413f710 2024-03-25T18:43:37.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.277+0000 7fe38bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:43:37.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.277+0000 7fe38bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:37.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.277+0000 7fe38bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:37.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.277+0000 7fe38bfff640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:43:37.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.277+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3940b28a0 con 0x7fe39413f710 2024-03-25T18:43:37.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1287837728 0 0) 0x7fe380003520 con 0x7fe3940a98f0 2024-03-25T18:43:37.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 >> v1:172.21.15.32:6790/0 conn(0x7fe39413f710 legacy=0x7fe3941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:37.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 >> v1:172.21.15.32:6789/0 conn(0x7fe3941430c0 legacy=0x7fe39414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:37.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 1 monclient: found mon.noname-c 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3940b44c0 con 0x7fe3940a98f0 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 10 monclient: _finish_auth 0 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe390001690 con 0x7fe3940a98f0 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe38bfff640 10 monclient: dump: 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.280 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.281 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:37.281 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:37.281 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:37.281 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:37.281 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:37.281 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:37.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe39b8ae640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:43:37.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.278+0000 7fe39b8ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3900017f0 con 0x7fe3940a98f0 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe38bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe38bfff640 1 -- 172.21.15.32:0/297400398 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe390001c00 con 0x7fe3940a98f0 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe39b8ae640 10 monclient: get_monmap_and_config success 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe39b8ae640 4 set_mon_vals no callback set 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe38bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe38bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe38bfff640 10 monclient: dump: 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:37.292 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.293 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.293 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:37.293 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:37.293 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:37.293 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:37.293 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:37.293 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:37.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe39b8ae640 10 monclient: shutdown 2024-03-25T18:43:37.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe39b8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:37.294 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.291+0000 7fe39b8ae640 1 -- 172.21.15.32:0/297400398 >> v1:172.21.15.136:6789/0 conn(0x7fe3940a98f0 legacy=0x7fe3940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.292+0000 7fe39b8ae640 1 -- 172.21.15.32:0/297400398 shutdown_connections 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.292+0000 7fe39b8ae640 1 -- 172.21.15.32:0/297400398 >> 172.21.15.32:0/297400398 conn(0x7fe3940b7ea0 msgr2=0x7fe3940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.292+0000 7fe39b8ae640 1 -- 172.21.15.32:0/297400398 shutdown_connections 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.292+0000 7fe39b8ae640 1 -- 172.21.15.32:0/297400398 wait complete. 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 10 monclient: build_initial_monmap 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 10 monclient: monmap: 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:37.295 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 1 librados: starting msgr at 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 1 librados: starting objecter 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 1 Processor -- start 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 1 -- start start 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 1 librados: setting wanted keys 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 1 librados: calling monclient init 2024-03-25T18:43:37.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.293+0000 7fe39b8ae640 10 monclient: init 2024-03-25T18:43:37.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:37.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:43:37.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 10 monclient(hunting): picked mon.noname-c con 0x7fe3940a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:37.297 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 10 monclient(hunting): picked mon.noname-b con 0x7fe39413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:37.298 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 10 monclient(hunting): picked mon.noname-a con 0x7fe3941430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:37.298 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe394139a40 con 0x7fe3941430c0 2024-03-25T18:43:37.298 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe394133f20 con 0x7fe39413f710 2024-03-25T18:43:37.298 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3940ab950 con 0x7fe3940a98f0 2024-03-25T18:43:37.298 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:37.298 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.294+0000 7fe39b8ae640 10 monclient(hunting): authenticate will time out at 6323.484074s 2024-03-25T18:43:37.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.299+0000 7fe398e49640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe39413f710 0x7fe394133100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:57898/0 (socket says 172.21.15.32:57898) 2024-03-25T18:43:37.300 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.299+0000 7fe398e49640 1 -- 172.21.15.32:0/2138180529 learned_addr learned my addr 172.21.15.32:0/2138180529 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:37.302 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.301+0000 7fe389ffb640 10 client.?.objecter ms_handle_connect 0x7fe39413f710 2024-03-25T18:43:37.302 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.301+0000 7fe389ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:37.306 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.305+0000 7fe389ffb640 10 client.?.objecter ms_handle_connect 0x7fe3941430c0 2024-03-25T18:43:37.306 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.305+0000 7fe389ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:37.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.309+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788620998 0 0) 0x7fe394133f20 con 0x7fe39413f710 2024-03-25T18:43:37.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.309+0000 7fe389ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:43:37.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.309+0000 7fe389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:37.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.309+0000 7fe389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:37.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.309+0000 7fe389ffb640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:43:37.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.309+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe378004eb0 con 0x7fe39413f710 2024-03-25T18:43:37.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.311+0000 7fe389ffb640 10 client.?.objecter ms_handle_connect 0x7fe3940a98f0 2024-03-25T18:43:37.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.311+0000 7fe389ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:37.316 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.315+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3212821887 0 0) 0x7fe394139a40 con 0x7fe3941430c0 2024-03-25T18:43:37.317 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.315+0000 7fe389ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:43:37.317 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.315+0000 7fe389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:37.317 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.315+0000 7fe389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:37.317 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.315+0000 7fe389ffb640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:43:37.317 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.315+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe394133f20 con 0x7fe3941430c0 2024-03-25T18:43:37.319 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.318+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 208622348 0 0) 0x7fe3940ab950 con 0x7fe3940a98f0 2024-03-25T18:43:37.319 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.318+0000 7fe389ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:43:37.319 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.318+0000 7fe389ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:37.319 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.318+0000 7fe389ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:37.320 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.318+0000 7fe389ffb640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:43:37.320 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.318+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe394139a40 con 0x7fe3940a98f0 2024-03-25T18:43:37.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.325+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2015602232 0 0) 0x7fe378004eb0 con 0x7fe39413f710 2024-03-25T18:43:37.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.325+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3940ab950 con 0x7fe39413f710 2024-03-25T18:43:37.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.325+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe384001a10 con 0x7fe39413f710 2024-03-25T18:43:37.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.325+0000 7fe389ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:37.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.325+0000 7fe389ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:43:37.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.325+0000 7fe389ffb640 10 monclient(hunting): dump: 2024-03-25T18:43:37.326 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:37.326 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:37.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.325+0000 7fe389ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:43:37.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.328+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 959524505 0 0) 0x7fe394139a40 con 0x7fe3940a98f0 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.328+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe378004eb0 con 0x7fe3940a98f0 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.328+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe390001b00 con 0x7fe3940a98f0 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.328+0000 7fe389ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.328+0000 7fe389ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.328+0000 7fe389ffb640 10 monclient(hunting): dump: 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:37.330 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:37.331 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:37.331 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:37.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.337+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562038470 0 0) 0x7fe394133f20 con 0x7fe3941430c0 2024-03-25T18:43:37.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.337+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe394139a40 con 0x7fe3941430c0 2024-03-25T18:43:37.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.337+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe38c001b10 con 0x7fe3941430c0 2024-03-25T18:43:37.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.337+0000 7fe389ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:37.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.337+0000 7fe389ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:43:37.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.337+0000 7fe389ffb640 10 monclient(hunting): dump: 2024-03-25T18:43:37.338 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:37.339 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:37.343 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 342315479 0 0) 0x7fe378004eb0 con 0x7fe3940a98f0 2024-03-25T18:43:37.343 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 >> v1:172.21.15.136:6789/0 conn(0x7fe39413f710 legacy=0x7fe394133100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:37.343 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 >> v1:172.21.15.32:6789/0 conn(0x7fe3941430c0 legacy=0x7fe394133810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:37.343 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:43:37.343 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 1 monclient: found mon.c 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3940abc30 con 0x7fe3940a98f0 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 10 monclient: _finish_auth 0 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe389ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe39b8ae640 5 monclient: authenticate success, global_id 4412 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe39b8ae640 10 monclient: _renew_subs 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe39b8ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.341+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3940abf10 con 0x7fe3940a98f0 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.342+0000 7fe39b8ae640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.342+0000 7fe39b8ae640 10 monclient: _renew_subs 2024-03-25T18:43:37.344 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.342+0000 7fe39b8ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:43:37.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.342+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe394139370 con 0x7fe3940a98f0 2024-03-25T18:43:37.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.342+0000 7fe39b8ae640 1 librados: init done 2024-03-25T18:43:37.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.343+0000 7fe39b8ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:37.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.343+0000 7fe39b8ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:37.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.343+0000 7fe39b8ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:43:37.345 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.343+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3940627b0 con 0x7fe3940a98f0 2024-03-25T18:43:37.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.661+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe390001c80 con 0x7fe3940a98f0 2024-03-25T18:43:37.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.662+0000 7fe389ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:37.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.662+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe390004c30 con 0x7fe3940a98f0 2024-03-25T18:43:37.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.662+0000 7fe389ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:37.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.662+0000 7fe389ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:43:37.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.662+0000 7fe389ffb640 10 monclient: dump: 2024-03-25T18:43:37.663 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:37.663 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:37.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.662+0000 7fe38b7fe640 4 set_mon_vals no callback set 2024-03-25T18:43:37.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.662+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fe39001ac00 con 0x7fe3940a98f0 2024-03-25T18:43:37.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.663+0000 7fe398e49640 10 monclient: get_auth_request con 0x7fe37806e110 auth_method 0 2024-03-25T18:43:37.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.663+0000 7fe389ffb640 10 client.4412.objecter ms_handle_connect 0x7fe37806e110 2024-03-25T18:43:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.669+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fe3900547c0 con 0x7fe3940a98f0 2024-03-25T18:43:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.669+0000 7fe389ffb640 10 client.4412.objecter ms_dispatch 0x7fe394000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:43:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.669+0000 7fe389ffb640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:43:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.669+0000 7fe389ffb640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:43:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.669+0000 7fe389ffb640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T18:43:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.669+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe3900502d0 con 0x7fe3940a98f0 2024-03-25T18:43:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.669+0000 7fe389ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.669+0000 7fe389ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:43:37.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.923+0000 7fe39b8ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:37.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.923+0000 7fe39b8ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:37.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.923+0000 7fe39b8ae640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:43:37.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.923+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3940b7180 con 0x7fe3940a98f0 2024-03-25T18:43:37.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.945+0000 7fe389ffb640 1 -- 172.21.15.32:0/2138180529 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fe390052200 con 0x7fe3940a98f0 2024-03-25T18:43:37.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.945+0000 7fe389ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:37.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.945+0000 7fe389ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:43:37.947 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:43:37.947 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:43:37.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.947+0000 7fe39b8ae640 10 librados: watch_flush enter 2024-03-25T18:43:37.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.947+0000 7fe39b8ae640 10 librados: watch_flush exit 2024-03-25T18:43:37.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.947+0000 7fe39b8ae640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T18:43:37.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.947+0000 7fe39b8ae640 10 client.4412.objecter successfully canceled tick 2024-03-25T18:43:37.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 >> v1:172.21.15.136:6800/1141502808 conn(0x7fe37806e110 legacy=0x7fe3780705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:37.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 10 monclient: shutdown 2024-03-25T18:43:37.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:37.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 >> v1:172.21.15.32:6790/0 conn(0x7fe3940a98f0 legacy=0x7fe394138c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:37.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 shutdown_connections 2024-03-25T18:43:37.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 >> 172.21.15.32:0/2138180529 conn(0x7fe3940b7ea0 msgr2=0x7fe394074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:37.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 shutdown_connections 2024-03-25T18:43:37.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 1 -- 172.21.15.32:0/2138180529 wait complete. 2024-03-25T18:43:37.950 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:37.948+0000 7fe39b8ae640 1 librados: shutdown 2024-03-25T18:43:37.962 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:43:44.770 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:43:44.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.885+0000 7fb623dcb640 10 monclient: get_monmap_and_config 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.885+0000 7fb623dcb640 10 monclient: build_initial_monmap 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.885+0000 7fb623dcb640 10 monclient: monmap: 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:43:44.887231+0000 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:43:44.887231+0000 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:43:44.887 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.886+0000 7fb623dcb640 1 Processor -- start 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.886+0000 7fb623dcb640 1 -- start start 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.886+0000 7fb623dcb640 10 monclient: init 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 10 monclient(hunting): picked mon.noname-c con 0x7fb61c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 10 monclient(hunting): picked mon.noname-b con 0x7fb61c13f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 10 monclient(hunting): picked mon.noname-a con 0x7fb61c1430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c142520 con 0x7fb61c1430c0 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c0b28a0 con 0x7fb61c0a98f0 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c0b2a30 con 0x7fb61c13f710 2024-03-25T18:43:44.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:44.890 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.887+0000 7fb623dcb640 10 monclient(hunting): authenticate will time out at 6331.077076s 2024-03-25T18:43:44.893 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.892+0000 7fb621366640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fb61c13f710 0x7fb61c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:48736/0 (socket says 172.21.15.32:48736) 2024-03-25T18:43:44.893 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.892+0000 7fb621366640 1 -- 172.21.15.32:0/2897081372 learned_addr learned my addr 172.21.15.32:0/2897081372 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:44.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.918+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3701186206 0 0) 0x7fb61c0b2a30 con 0x7fb61c13f710 2024-03-25T18:43:44.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.918+0000 7fb620b65640 10 monclient(hunting): _negotiate 2024-03-25T18:43:44.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.918+0000 7fb620b65640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:44.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.918+0000 7fb620b65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:44.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.918+0000 7fb620b65640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:43:44.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.918+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb608003520 con 0x7fb61c13f710 2024-03-25T18:43:44.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.922+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438704480 0 0) 0x7fb61c0b28a0 con 0x7fb61c0a98f0 2024-03-25T18:43:44.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.922+0000 7fb620b65640 10 monclient(hunting): _negotiate 2024-03-25T18:43:44.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.922+0000 7fb620b65640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:44.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.922+0000 7fb620b65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:44.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.922+0000 7fb620b65640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:43:44.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.922+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb61c0b2a30 con 0x7fb61c0a98f0 2024-03-25T18:43:44.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.935+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1667379049 0 0) 0x7fb61c142520 con 0x7fb61c1430c0 2024-03-25T18:43:44.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.935+0000 7fb620b65640 10 monclient(hunting): _negotiate 2024-03-25T18:43:44.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.935+0000 7fb620b65640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:44.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.935+0000 7fb620b65640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:44.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.935+0000 7fb620b65640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:43:44.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.935+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb61c0b28a0 con 0x7fb61c1430c0 2024-03-25T18:43:44.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2451904659 0 0) 0x7fb608003520 con 0x7fb61c13f710 2024-03-25T18:43:44.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 >> v1:172.21.15.136:6789/0 conn(0x7fb61c0a98f0 legacy=0x7fb61c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:44.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 >> v1:172.21.15.32:6789/0 conn(0x7fb61c1430c0 legacy=0x7fb61c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:44.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 10 monclient: _finish_hunting 0 2024-03-25T18:43:44.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 1 monclient: found mon.noname-b 2024-03-25T18:43:44.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c0b44c0 con 0x7fb61c13f710 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 10 monclient: _finish_auth 0 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 10 monclient: _check_auth_tickets 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb60c001690 con 0x7fb61c13f710 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb620b65640 10 monclient: dump: 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:44.940 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:44.941 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:44.941 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:44.941 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:44.941 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:44.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb623dcb640 5 monclient: authenticate success, global_id 4415 2024-03-25T18:43:44.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.938+0000 7fb623dcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:43:44.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.970+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb60c0017f0 con 0x7fb61c13f710 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.970+0000 7fb620b65640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.970+0000 7fb620b65640 1 -- 172.21.15.32:0/2897081372 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb60c001c00 con 0x7fb61c13f710 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.970+0000 7fb623dcb640 10 monclient: get_monmap_and_config success 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.970+0000 7fb623dcb640 4 set_mon_vals no callback set 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.970+0000 7fb620b65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.970+0000 7fb620b65640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.970+0000 7fb620b65640 10 monclient: dump: 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:44.973 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.971+0000 7fb623dcb640 10 monclient: shutdown 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.971+0000 7fb623dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.971+0000 7fb623dcb640 1 -- 172.21.15.32:0/2897081372 >> v1:172.21.15.32:6790/0 conn(0x7fb61c13f710 legacy=0x7fb61c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.971+0000 7fb623dcb640 1 -- 172.21.15.32:0/2897081372 shutdown_connections 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.971+0000 7fb623dcb640 1 -- 172.21.15.32:0/2897081372 >> 172.21.15.32:0/2897081372 conn(0x7fb61c0b7ea0 msgr2=0x7fb61c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.971+0000 7fb623dcb640 1 -- 172.21.15.32:0/2897081372 shutdown_connections 2024-03-25T18:43:44.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.971+0000 7fb623dcb640 1 -- 172.21.15.32:0/2897081372 wait complete. 2024-03-25T18:43:44.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.972+0000 7fb623dcb640 10 monclient: build_initial_monmap 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.972+0000 7fb623dcb640 10 monclient: monmap: 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.973+0000 7fb623dcb640 1 librados: starting msgr at 2024-03-25T18:43:44.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.973+0000 7fb623dcb640 1 librados: starting objecter 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.973+0000 7fb623dcb640 1 Processor -- start 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.973+0000 7fb623dcb640 1 -- start start 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.973+0000 7fb623dcb640 1 librados: setting wanted keys 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.973+0000 7fb623dcb640 1 librados: calling monclient init 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.973+0000 7fb623dcb640 10 monclient: init 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 10 monclient(hunting): picked mon.noname-a con 0x7fb61c0a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 10 monclient(hunting): picked mon.noname-c con 0x7fb61c13f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 10 monclient(hunting): picked mon.noname-b con 0x7fb61c1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c0aba00 con 0x7fb61c0a98f0 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c1d7490 con 0x7fb61c1430c0 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb61c1d8360 con 0x7fb61c13f710 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:44.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.974+0000 7fb623dcb640 10 monclient(hunting): authenticate will time out at 6331.163711s 2024-03-25T18:43:44.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.986+0000 7fb622368640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb61c1430c0 0x7fb61c133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:34750/0 (socket says 172.21.15.32:34750) 2024-03-25T18:43:44.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.986+0000 7fb622368640 1 -- 172.21.15.32:0/1010275828 learned_addr learned my addr 172.21.15.32:0/1010275828 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:44.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.987+0000 7fb6127fc640 10 client.?.objecter ms_handle_connect 0x7fb61c1430c0 2024-03-25T18:43:44.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.987+0000 7fb6127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:44.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.995+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937295923 0 0) 0x7fb61c1d7490 con 0x7fb61c1430c0 2024-03-25T18:43:44.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.995+0000 7fb6127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:43:44.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.995+0000 7fb6127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:44.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.995+0000 7fb6127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:44.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.995+0000 7fb6127fc640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:43:44.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.996+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb600004eb0 con 0x7fb61c1430c0 2024-03-25T18:43:44.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.996+0000 7fb6127fc640 10 client.?.objecter ms_handle_connect 0x7fb61c13f710 2024-03-25T18:43:44.997 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.996+0000 7fb6127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:44.999 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.998+0000 7fb6127fc640 10 client.?.objecter ms_handle_connect 0x7fb61c0a98f0 2024-03-25T18:43:44.999 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:44.998+0000 7fb6127fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:45.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.008+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2178066904 0 0) 0x7fb61c0aba00 con 0x7fb61c0a98f0 2024-03-25T18:43:45.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.008+0000 7fb6127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:43:45.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.008+0000 7fb6127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:45.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.008+0000 7fb6127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:45.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.008+0000 7fb6127fc640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:43:45.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.008+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb61c1d7490 con 0x7fb61c0a98f0 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.009+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324408539 0 0) 0x7fb600004eb0 con 0x7fb61c1430c0 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.009+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb61c0aba00 con 0x7fb61c1430c0 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.009+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb614001c70 con 0x7fb61c1430c0 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.009+0000 7fb6127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.009+0000 7fb6127fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.009+0000 7fb6127fc640 10 monclient(hunting): dump: 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:45.011 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:45.012 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:45.012 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:45.012 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:45.012 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:45.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.009+0000 7fb6127fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:43:45.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.014+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 890168525 0 0) 0x7fb61c1d8360 con 0x7fb61c13f710 2024-03-25T18:43:45.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.014+0000 7fb6127fc640 10 monclient(hunting): _negotiate 2024-03-25T18:43:45.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.014+0000 7fb6127fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:45.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.014+0000 7fb6127fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:45.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.014+0000 7fb6127fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:43:45.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.014+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb600004eb0 con 0x7fb61c13f710 2024-03-25T18:43:45.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 310294563 0 0) 0x7fb61c0aba00 con 0x7fb61c1430c0 2024-03-25T18:43:45.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 >> v1:172.21.15.32:6790/0 conn(0x7fb61c13f710 legacy=0x7fb61c1d7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:45.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 >> v1:172.21.15.32:6789/0 conn(0x7fb61c0a98f0 legacy=0x7fb61c1386e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:45.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 10 monclient: _finish_hunting 0 2024-03-25T18:43:45.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 1 monclient: found mon.b 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb61c0abc30 con 0x7fb61c1430c0 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 10 monclient: _finish_auth 0 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 10 monclient: _check_auth_tickets 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb6127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.023+0000 7fb623dcb640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.024+0000 7fb623dcb640 10 monclient: _renew_subs 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.024+0000 7fb623dcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.024+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb61c137940 con 0x7fb61c1430c0 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.024+0000 7fb623dcb640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.024+0000 7fb623dcb640 10 monclient: _renew_subs 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.024+0000 7fb623dcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:45.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.024+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb61c135220 con 0x7fb61c1430c0 2024-03-25T18:43:45.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.024+0000 7fb623dcb640 1 librados: init done 2024-03-25T18:43:45.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.025+0000 7fb623dcb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:45.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.025+0000 7fb623dcb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:45.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.025+0000 7fb623dcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:45.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.025+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb61c0627b0 con 0x7fb61c1430c0 2024-03-25T18:43:45.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.302+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb614003990 con 0x7fb61c1430c0 2024-03-25T18:43:45.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.302+0000 7fb6127fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:45.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.302+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb614003da0 con 0x7fb61c1430c0 2024-03-25T18:43:45.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.302+0000 7fb6127fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:45.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.303+0000 7fb6127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:45.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.303+0000 7fb6127fc640 10 monclient: dump: 2024-03-25T18:43:45.304 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:45.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.303+0000 7fb613fff640 4 set_mon_vals no callback set 2024-03-25T18:43:45.306 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.303+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fb614003fc0 con 0x7fb61c1430c0 2024-03-25T18:43:45.306 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.304+0000 7fb621b67640 10 monclient: get_auth_request con 0x7fb60006de00 auth_method 0 2024-03-25T18:43:45.306 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.304+0000 7fb6127fc640 10 client.4414.objecter ms_handle_connect 0x7fb60006de00 2024-03-25T18:43:45.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.308+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fb614054ae0 con 0x7fb61c1430c0 2024-03-25T18:43:45.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.308+0000 7fb6127fc640 10 client.4414.objecter ms_dispatch 0x7fb61c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:43:45.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.308+0000 7fb6127fc640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:43:45.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.308+0000 7fb6127fc640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:43:45.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.309+0000 7fb6127fc640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T18:43:45.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.309+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6140505f0 con 0x7fb61c1430c0 2024-03-25T18:43:45.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.309+0000 7fb6127fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:45.310 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.309+0000 7fb6127fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:43:45.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.541+0000 7fb623dcb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:45.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.541+0000 7fb623dcb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:45.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.541+0000 7fb623dcb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:45.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.541+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb61c069f40 con 0x7fb61c1430c0 2024-03-25T18:43:45.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.580+0000 7fb6127fc640 1 -- 172.21.15.32:0/1010275828 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fb614052520 con 0x7fb61c1430c0 2024-03-25T18:43:45.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.580+0000 7fb6127fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:45.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.580+0000 7fb6127fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:43:45.581 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:43:45.582 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:43:45.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 10 librados: watch_flush enter 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 10 librados: watch_flush exit 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 10 client.4414.objecter successfully canceled tick 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 >> v1:172.21.15.136:6800/1141502808 conn(0x7fb60006de00 legacy=0x7fb6000702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 10 monclient: shutdown 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 >> v1:172.21.15.136:6789/0 conn(0x7fb61c1430c0 legacy=0x7fb61c133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 shutdown_connections 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 >> 172.21.15.32:0/1010275828 conn(0x7fb61c0b7ea0 msgr2=0x7fb61c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 shutdown_connections 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 1 -- 172.21.15.32:0/1010275828 wait complete. 2024-03-25T18:43:45.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:45.582+0000 7fb623dcb640 1 librados: shutdown 2024-03-25T18:43:45.595 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:43:47.750 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:00.078 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:47.763 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:00.091 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:47.786 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:00.114 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:47.806 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:00.123 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:47.811 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:00.128 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:47.811 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:00.139 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:47.815 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:00.143 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:47.850 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:00.167 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:47.856 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:00.184 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:43:52.401 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:43:52.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.510+0000 7f012c2b5640 10 monclient: get_monmap_and_config 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.510+0000 7f012c2b5640 10 monclient: build_initial_monmap 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.510+0000 7f012c2b5640 10 monclient: monmap: 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:43:52.510711+0000 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:43:52.510711+0000 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.510+0000 7f012c2b5640 1 Processor -- start 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.510+0000 7f012c2b5640 1 -- start start 2024-03-25T18:43:52.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.510+0000 7f012c2b5640 10 monclient: init 2024-03-25T18:43:52.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 10 monclient(hunting): picked mon.noname-c con 0x7f01240a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 10 monclient(hunting): picked mon.noname-b con 0x7f012413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 10 monclient(hunting): picked mon.noname-a con 0x7f01241430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0124142520 con 0x7f01241430c0 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01240b28a0 con 0x7f01240a98f0 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01240b2a30 con 0x7f012413f710 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.511+0000 7f012c2b5640 10 monclient(hunting): authenticate will time out at 6338.700203s 2024-03-25T18:43:52.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.513+0000 7f0129850640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f012413f710 0x7f01241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:35146/0 (socket says 172.21.15.32:35146) 2024-03-25T18:43:52.514 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.513+0000 7f0129850640 1 -- 172.21.15.32:0/2518032307 learned_addr learned my addr 172.21.15.32:0/2518032307 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:52.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.583+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190687369 0 0) 0x7f01240b2a30 con 0x7f012413f710 2024-03-25T18:43:52.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.583+0000 7f012904f640 10 monclient(hunting): _negotiate 2024-03-25T18:43:52.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.583+0000 7f012904f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:52.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.583+0000 7f012904f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:52.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.583+0000 7f012904f640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:43:52.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.583+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110003520 con 0x7f012413f710 2024-03-25T18:43:52.586 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.586+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1468012708 0 0) 0x7f01240b28a0 con 0x7f01240a98f0 2024-03-25T18:43:52.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.586+0000 7f012904f640 10 monclient(hunting): _negotiate 2024-03-25T18:43:52.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.586+0000 7f012904f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:52.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.586+0000 7f012904f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:52.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.586+0000 7f012904f640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:43:52.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.586+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01240b2a30 con 0x7f01240a98f0 2024-03-25T18:43:52.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3484380979 0 0) 0x7f0110003520 con 0x7f012413f710 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 >> v1:172.21.15.136:6789/0 conn(0x7f01240a98f0 legacy=0x7f01240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 >> v1:172.21.15.32:6789/0 conn(0x7f01241430c0 legacy=0x7f012414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 10 monclient: _finish_hunting 0 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 1 monclient: found mon.noname-b 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01240b44c0 con 0x7f012413f710 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 10 monclient: _finish_auth 0 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 10 monclient: _check_auth_tickets 2024-03-25T18:43:52.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0114001690 con 0x7f012413f710 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012904f640 10 monclient: dump: 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:52.598 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:52.599 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:52.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012c2b5640 5 monclient: authenticate success, global_id 4421 2024-03-25T18:43:52.599 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.596+0000 7f012c2b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:43:52.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.610+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01140017f0 con 0x7f012413f710 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.610+0000 7f012904f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.610+0000 7f012904f640 1 -- 172.21.15.32:0/2518032307 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0114001c00 con 0x7f012413f710 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.610+0000 7f012c2b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.610+0000 7f012c2b5640 4 set_mon_vals no callback set 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.610+0000 7f012904f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.610+0000 7f012904f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.610+0000 7f012904f640 10 monclient: dump: 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:52.611 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:52.612 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:52.612 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:52.612 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:52.612 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.611+0000 7f012c2b5640 10 monclient: shutdown 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.611+0000 7f012c2b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.611+0000 7f012c2b5640 1 -- 172.21.15.32:0/2518032307 >> v1:172.21.15.32:6790/0 conn(0x7f012413f710 legacy=0x7f01241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.611+0000 7f012c2b5640 1 -- 172.21.15.32:0/2518032307 shutdown_connections 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.611+0000 7f012c2b5640 1 -- 172.21.15.32:0/2518032307 >> 172.21.15.32:0/2518032307 conn(0x7f01240b7ea0 msgr2=0x7f01240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.611+0000 7f012c2b5640 1 -- 172.21.15.32:0/2518032307 shutdown_connections 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.611+0000 7f012c2b5640 1 -- 172.21.15.32:0/2518032307 wait complete. 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 10 monclient: build_initial_monmap 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 10 monclient: monmap: 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:43:52.614 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 1 librados: starting msgr at 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 1 librados: starting objecter 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 1 Processor -- start 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 1 -- start start 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 1 librados: setting wanted keys 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 1 librados: calling monclient init 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.612+0000 7f012c2b5640 10 monclient: init 2024-03-25T18:43:52.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 10 monclient(hunting): picked mon.noname-c con 0x7f01240a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 10 monclient(hunting): picked mon.noname-a con 0x7f012413f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 10 monclient(hunting): picked mon.noname-b con 0x7f01241430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0124138b20 con 0x7f012413f710 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0124137a80 con 0x7f01241430c0 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0124139880 con 0x7f01240a98f0 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:43:52.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.613+0000 7f012c2b5640 10 monclient(hunting): authenticate will time out at 6338.802028s 2024-03-25T18:43:52.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.617+0000 7f012a852640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f01241430c0 0x7f01241e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:60556/0 (socket says 172.21.15.32:60556) 2024-03-25T18:43:52.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.617+0000 7f012a852640 1 -- 172.21.15.32:0/2779183118 learned_addr learned my addr 172.21.15.32:0/2779183118 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:43:52.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.621+0000 7f011affd640 10 client.?.objecter ms_handle_connect 0x7f01241430c0 2024-03-25T18:43:52.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.621+0000 7f011affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:52.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.622+0000 7f011affd640 10 client.?.objecter ms_handle_connect 0x7f012413f710 2024-03-25T18:43:52.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.622+0000 7f011affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:52.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.625+0000 7f011affd640 10 client.?.objecter ms_handle_connect 0x7f01240a98f0 2024-03-25T18:43:52.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.625+0000 7f011affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:43:52.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.628+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3262759878 0 0) 0x7f0124137a80 con 0x7f01241430c0 2024-03-25T18:43:52.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.628+0000 7f011affd640 10 monclient(hunting): _negotiate 2024-03-25T18:43:52.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.628+0000 7f011affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:52.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.628+0000 7f011affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:52.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.628+0000 7f011affd640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:43:52.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.628+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0108004eb0 con 0x7f01241430c0 2024-03-25T18:43:52.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.633+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403078646 0 0) 0x7f0124138b20 con 0x7f012413f710 2024-03-25T18:43:52.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.633+0000 7f011affd640 10 monclient(hunting): _negotiate 2024-03-25T18:43:52.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.633+0000 7f011affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:52.633 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.633+0000 7f011affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:52.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.633+0000 7f011affd640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:43:52.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.633+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0124137a80 con 0x7f012413f710 2024-03-25T18:43:52.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1692346744 0 0) 0x7f0124139880 con 0x7f01240a98f0 2024-03-25T18:43:52.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 10 monclient(hunting): _negotiate 2024-03-25T18:43:52.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:43:52.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:43:52.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:43:52.636 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0124138b20 con 0x7f01240a98f0 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2848592170 0 0) 0x7f0108004eb0 con 0x7f01241430c0 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0124139880 con 0x7f01241430c0 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f011c001c90 con 0x7f01241430c0 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 10 monclient(hunting): dump: 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.637 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.638 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:52.638 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:52.638 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:52.638 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:52.638 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:52.638 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:52.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.636+0000 7f011affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2691773358 0 0) 0x7f0124137a80 con 0x7f012413f710 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0108004eb0 con 0x7f012413f710 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0114001b00 con 0x7f012413f710 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 10 monclient(hunting): dump: 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.647 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 20842304 0 0) 0x7f0124139880 con 0x7f01241430c0 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 >> v1:172.21.15.32:6790/0 conn(0x7f01240a98f0 legacy=0x7f0124137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 >> v1:172.21.15.32:6789/0 conn(0x7f012413f710 legacy=0x7f0124139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 10 monclient: _finish_hunting 0 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 1 monclient: found mon.b 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:52.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124139a10 con 0x7f01241430c0 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 10 monclient: _finish_auth 0 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 10 monclient: _check_auth_tickets 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f011affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 10 monclient: _renew_subs 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01241dab00 con 0x7f01241430c0 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 10 monclient: _renew_subs 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f012413b4f0 con 0x7f01241430c0 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.646+0000 7f012c2b5640 1 librados: init done 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.647+0000 7f012c2b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.647+0000 7f012c2b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:52.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.647+0000 7f012c2b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:52.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.647+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01240b3e80 con 0x7f01241430c0 2024-03-25T18:43:52.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.916+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f011c003990 con 0x7f01241430c0 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.916+0000 7f011affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.916+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f011c003ce0 con 0x7f01241430c0 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.916+0000 7f011affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.916+0000 7f011affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.916+0000 7f011affd640 10 monclient: dump: 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:43:52.918 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:43:52.919 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:43:52.919 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:43:52.919 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:43:52.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.916+0000 7f012884e640 4 set_mon_vals no callback set 2024-03-25T18:43:52.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.917+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f011c003f00 con 0x7f01241430c0 2024-03-25T18:43:52.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.918+0000 7f012a051640 10 monclient: get_auth_request con 0x7f010806de70 auth_method 0 2024-03-25T18:43:52.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.918+0000 7f011affd640 10 client.4420.objecter ms_handle_connect 0x7f010806de70 2024-03-25T18:43:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.923+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f011c054ae0 con 0x7f01241430c0 2024-03-25T18:43:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.923+0000 7f011affd640 10 client.4420.objecter ms_dispatch 0x7f0124000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:43:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.923+0000 7f011affd640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:43:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.923+0000 7f011affd640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:43:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.923+0000 7f011affd640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T18:43:52.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.923+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f011c0505f0 con 0x7f01241430c0 2024-03-25T18:43:52.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.923+0000 7f011affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:43:52.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:52.923+0000 7f011affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:43:53.172 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.172+0000 7f012c2b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:53.172 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.172+0000 7f012c2b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:53.172 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.172+0000 7f012c2b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:43:53.172 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.172+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f01240627b0 con 0x7f01241430c0 2024-03-25T18:43:53.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.195+0000 7f011affd640 1 -- 172.21.15.32:0/2779183118 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f011c052520 con 0x7f01241430c0 2024-03-25T18:43:53.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.195+0000 7f011affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:43:53.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.195+0000 7f011affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:43:53.195 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:43:53.195 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:43:53.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 10 librados: watch_flush enter 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 10 librados: watch_flush exit 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 10 client.4420.objecter successfully canceled tick 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 >> v1:172.21.15.136:6800/1141502808 conn(0x7f010806de70 legacy=0x7f0108070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 10 monclient: shutdown 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 >> v1:172.21.15.136:6789/0 conn(0x7f01241430c0 legacy=0x7f01241e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 shutdown_connections 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 >> 172.21.15.32:0/2779183118 conn(0x7f01240b7ea0 msgr2=0x7f0124074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 shutdown_connections 2024-03-25T18:43:53.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 1 -- 172.21.15.32:0/2779183118 wait complete. 2024-03-25T18:43:53.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:43:53.197+0000 7f012c2b5640 1 librados: shutdown 2024-03-25T18:43:53.210 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:44:00.017 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:44:00.137 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.133+0000 7f85d5e55640 10 monclient: get_monmap_and_config 2024-03-25T18:44:00.137 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.133+0000 7f85d5e55640 10 monclient: build_initial_monmap 2024-03-25T18:44:00.137 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.133+0000 7f85d5e55640 10 monclient: monmap: 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:44:00.133632+0000 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:44:00.133632+0000 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.133+0000 7f85d5e55640 1 Processor -- start 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.134+0000 7f85d5e55640 1 -- start start 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.134+0000 7f85d5e55640 10 monclient: init 2024-03-25T18:44:00.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.134+0000 7f85d5e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.134+0000 7f85d5e55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.134+0000 7f85d5e55640 10 monclient(hunting): picked mon.noname-c con 0x7f85d00b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.134+0000 7f85d5e55640 10 monclient(hunting): picked mon.noname-b con 0x7f85d00b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.134+0000 7f85d5e55640 10 monclient(hunting): picked mon.noname-a con 0x7f85d014be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.135+0000 7f85d5e55640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d00af940 con 0x7f85d014be90 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.135+0000 7f85d5e55640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d00b0a70 con 0x7f85d00b4d60 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.135+0000 7f85d5e55640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d00b0d50 con 0x7f85d00b1730 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.135+0000 7f85d5e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:00.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.135+0000 7f85d5e55640 10 monclient(hunting): authenticate will time out at 6346.323611s 2024-03-25T18:44:00.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.145+0000 7f85cf7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f85d00b4d60 0x7f85d013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:46384/0 (socket says 172.21.15.32:46384) 2024-03-25T18:44:00.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.145+0000 7f85cf7fe640 1 -- 172.21.15.32:0/3324238274 learned_addr learned my addr 172.21.15.32:0/3324238274 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:00.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.151+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371658870 0 0) 0x7f85d00b0a70 con 0x7f85d00b4d60 2024-03-25T18:44:00.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.151+0000 7f85ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:44:00.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.151+0000 7f85ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:00.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.151+0000 7f85ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:00.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.151+0000 7f85ce7fc640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:44:00.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.151+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85b8003520 con 0x7f85d00b4d60 2024-03-25T18:44:00.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 613177152 0 0) 0x7f85b8003520 con 0x7f85d00b4d60 2024-03-25T18:44:00.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 >> v1:172.21.15.32:6790/0 conn(0x7f85d00b1730 legacy=0x7f85d00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:00.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 >> v1:172.21.15.32:6789/0 conn(0x7f85d014be90 legacy=0x7f85d014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:00.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 1 monclient: found mon.noname-c 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d00b1030 con 0x7f85d00b4d60 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 10 monclient: _finish_auth 0 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f85c0001690 con 0x7f85d00b4d60 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85ce7fc640 10 monclient: dump: 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:00.169 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85d5e55640 5 monclient: authenticate success, global_id 4423 2024-03-25T18:44:00.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.168+0000 7f85d5e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.176+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85c00017f0 con 0x7f85d00b4d60 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.176+0000 7f85ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.176+0000 7f85ce7fc640 1 -- 172.21.15.32:0/3324238274 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f85c0001c00 con 0x7f85d00b4d60 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.176+0000 7f85ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.176+0000 7f85ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.176+0000 7f85ce7fc640 10 monclient: dump: 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:00.176 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:00.177 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:00.177 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:00.177 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:00.177 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:00.177 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:00.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.176+0000 7f85d5e55640 10 monclient: get_monmap_and_config success 2024-03-25T18:44:00.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.176+0000 7f85d5e55640 4 set_mon_vals no callback set 2024-03-25T18:44:00.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.177+0000 7f85d5e55640 10 monclient: shutdown 2024-03-25T18:44:00.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.177+0000 7f85d5e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:00.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.177+0000 7f85d5e55640 1 -- 172.21.15.32:0/3324238274 >> v1:172.21.15.136:6789/0 conn(0x7f85d00b4d60 legacy=0x7f85d013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:00.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.177+0000 7f85d5e55640 1 -- 172.21.15.32:0/3324238274 shutdown_connections 2024-03-25T18:44:00.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.177+0000 7f85d5e55640 1 -- 172.21.15.32:0/3324238274 >> 172.21.15.32:0/3324238274 conn(0x7f85d00617a0 msgr2=0x7f85d0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.177+0000 7f85d5e55640 1 -- 172.21.15.32:0/3324238274 shutdown_connections 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.177+0000 7f85d5e55640 1 -- 172.21.15.32:0/3324238274 wait complete. 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.178+0000 7f85d5e55640 10 monclient: build_initial_monmap 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.178+0000 7f85d5e55640 10 monclient: monmap: 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:00.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.178+0000 7f85d5e55640 1 librados: starting msgr at 2024-03-25T18:44:00.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.178+0000 7f85d5e55640 1 librados: starting objecter 2024-03-25T18:44:00.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.178+0000 7f85d5e55640 1 Processor -- start 2024-03-25T18:44:00.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 1 -- start start 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 1 librados: setting wanted keys 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 1 librados: calling monclient init 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 10 monclient: init 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 10 monclient(hunting): picked mon.noname-b con 0x7f85d00b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 10 monclient(hunting): picked mon.noname-a con 0x7f85d00b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 10 monclient(hunting): picked mon.noname-c con 0x7f85d014be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d0141f40 con 0x7f85d00b4d60 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d0142360 con 0x7f85d00b1730 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85d01424f0 con 0x7f85d014be90 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:00.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.179+0000 7f85d5e55640 10 monclient(hunting): authenticate will time out at 6346.368292s 2024-03-25T18:44:00.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.196+0000 7f85cffff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f85d014be90 0x7f85d01e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:37684/0 (socket says 172.21.15.32:37684) 2024-03-25T18:44:00.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.196+0000 7f85cffff640 1 -- 172.21.15.32:0/4118075521 learned_addr learned my addr 172.21.15.32:0/4118075521 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:00.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.198+0000 7f85b7fff640 10 client.?.objecter ms_handle_connect 0x7f85d00b1730 2024-03-25T18:44:00.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.198+0000 7f85b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:00.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.198+0000 7f85b7fff640 10 client.?.objecter ms_handle_connect 0x7f85d014be90 2024-03-25T18:44:00.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.198+0000 7f85b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:00.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.200+0000 7f85b7fff640 10 client.?.objecter ms_handle_connect 0x7f85d00b4d60 2024-03-25T18:44:00.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.200+0000 7f85b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:00.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.209+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1347640377 0 0) 0x7f85d0142360 con 0x7f85d00b1730 2024-03-25T18:44:00.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.209+0000 7f85b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:44:00.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.209+0000 7f85b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:00.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.209+0000 7f85b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:00.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.209+0000 7f85b7fff640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:44:00.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.209+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85bc004eb0 con 0x7f85d00b1730 2024-03-25T18:44:00.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.219+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2558338527 0 0) 0x7f85d00af940 con 0x7f85d014be90 2024-03-25T18:44:00.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.219+0000 7f85b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:44:00.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.219+0000 7f85b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:00.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.219+0000 7f85b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:00.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.219+0000 7f85b7fff640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:44:00.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.219+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d0142360 con 0x7f85d014be90 2024-03-25T18:44:00.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.224+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158034245 0 0) 0x7f85d0141f40 con 0x7f85d00b4d60 2024-03-25T18:44:00.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.224+0000 7f85b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:44:00.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.224+0000 7f85b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:00.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.224+0000 7f85b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:00.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.224+0000 7f85b7fff640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:44:00.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.224+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d00af940 con 0x7f85d00b4d60 2024-03-25T18:44:00.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.240+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1036834509 0 0) 0x7f85d00af940 con 0x7f85d00b4d60 2024-03-25T18:44:00.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.240+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f85d0141f40 con 0x7f85d00b4d60 2024-03-25T18:44:00.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.240+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f85c4001830 con 0x7f85d00b4d60 2024-03-25T18:44:00.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.240+0000 7f85b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.240+0000 7f85b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.240+0000 7f85b7fff640 10 monclient(hunting): dump: 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:00.241 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:00.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.240+0000 7f85b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:44:00.311 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2060103237 0 0) 0x7f85d0141f40 con 0x7f85d00b4d60 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 >> v1:172.21.15.32:6790/0 conn(0x7f85d014be90 legacy=0x7f85d01e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 >> v1:172.21.15.136:6789/0 conn(0x7f85d00b1730 legacy=0x7f85d01405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 1 monclient: found mon.a 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d0140d00 con 0x7f85d00b4d60 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 5 monclient: authenticate success, global_id 4443 2024-03-25T18:44:00.312 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 10 monclient: _renew_subs 2024-03-25T18:44:00.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:00.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85d0138150 con 0x7f85d00b4d60 2024-03-25T18:44:00.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 10 client.4443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:44:00.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 10 monclient: _renew_subs 2024-03-25T18:44:00.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:00.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f85d01d8520 con 0x7f85d00b4d60 2024-03-25T18:44:00.313 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.311+0000 7f85d5e55640 1 librados: init done 2024-03-25T18:44:00.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.312+0000 7f85d5e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:00.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.312+0000 7f85d5e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:00.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.312+0000 7f85d5e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:00.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.312+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85d0140f50 con 0x7f85d00b4d60 2024-03-25T18:44:00.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.645+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85c4001d40 con 0x7f85d00b4d60 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.645+0000 7f85b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.645+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f85c4003790 con 0x7f85d00b4d60 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.645+0000 7f85b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.645+0000 7f85b7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.645+0000 7f85b7fff640 10 monclient: dump: 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:00.646 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:00.647 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:00.647 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:00.647 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:00.647 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:00.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.645+0000 7f85cdffb640 4 set_mon_vals no callback set 2024-03-25T18:44:00.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.646+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f85c40039b0 con 0x7f85d00b4d60 2024-03-25T18:44:00.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.647+0000 7f85cf7fe640 10 monclient: get_auth_request con 0x7f85bc06ddd0 auth_method 0 2024-03-25T18:44:00.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.647+0000 7f85b7fff640 10 client.4443.objecter ms_handle_connect 0x7f85bc06ddd0 2024-03-25T18:44:00.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.651+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f85c4053590 con 0x7f85d00b4d60 2024-03-25T18:44:00.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.651+0000 7f85b7fff640 10 client.4443.objecter ms_dispatch 0x7f85d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:44:00.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.651+0000 7f85b7fff640 3 client.4443.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:44:00.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.651+0000 7f85b7fff640 3 client.4443.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:44:00.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.651+0000 7f85b7fff640 20 client.4443.objecter dump_active .. 0 homeless 2024-03-25T18:44:00.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.651+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85c404eed0 con 0x7f85d00b4d60 2024-03-25T18:44:00.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.651+0000 7f85b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:00.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.651+0000 7f85b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:44:00.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.909+0000 7f85d5e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:00.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.909+0000 7f85d5e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:00.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.909+0000 7f85d5e55640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:00.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.909+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85d01d7950 con 0x7f85d00b4d60 2024-03-25T18:44:00.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.941+0000 7f85b7fff640 1 -- 172.21.15.32:0/4118075521 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f85d01d7950 con 0x7f85d00b4d60 2024-03-25T18:44:00.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.941+0000 7f85b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:00.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.941+0000 7f85b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:44:00.942 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:44:00.942 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:44:00.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.943+0000 7f85d5e55640 10 librados: watch_flush enter 2024-03-25T18:44:00.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 10 librados: watch_flush exit 2024-03-25T18:44:00.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 20 client.4443.objecter shutdown clearing up homeless session... 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 10 client.4443.objecter successfully canceled tick 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 >> v1:172.21.15.136:6800/1141502808 conn(0x7f85bc06ddd0 legacy=0x7f85bc070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 10 monclient: shutdown 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 >> v1:172.21.15.32:6789/0 conn(0x7f85d00b4d60 legacy=0x7f85d01d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 shutdown_connections 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 >> 172.21.15.32:0/4118075521 conn(0x7f85d00617a0 msgr2=0x7f85d00b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 shutdown_connections 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 1 -- 172.21.15.32:0/4118075521 wait complete. 2024-03-25T18:44:00.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:00.944+0000 7f85d5e55640 1 librados: shutdown 2024-03-25T18:44:00.956 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:44:07.764 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:44:07.830 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:20.158 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.836 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:20.164 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.859+0000 7f8d263af640 10 monclient: get_monmap_and_config 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.859+0000 7f8d263af640 10 monclient: build_initial_monmap 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.859+0000 7f8d263af640 10 monclient: monmap: 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:44:07.860297+0000 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:44:07.860297+0000 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:44:07.860 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:07.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.860+0000 7f8d263af640 1 Processor -- start 2024-03-25T18:44:07.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.860+0000 7f8d263af640 1 -- start start 2024-03-25T18:44:07.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.860+0000 7f8d263af640 10 monclient: init 2024-03-25T18:44:07.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:07.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:44:07.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 10 monclient(hunting): picked mon.noname-b con 0x7f8d200a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:07.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 10 monclient(hunting): picked mon.noname-a con 0x7f8d2013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:07.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 10 monclient(hunting): picked mon.noname-c con 0x7f8d201430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:07.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d20142520 con 0x7f8d2013f710 2024-03-25T18:44:07.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d200b28a0 con 0x7f8d201430c0 2024-03-25T18:44:07.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d200b2a30 con 0x7f8d200a98f0 2024-03-25T18:44:07.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:07.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.861+0000 7f8d263af640 10 monclient(hunting): authenticate will time out at 6354.049793s 2024-03-25T18:44:07.870 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:20.198 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.891+0000 7f8d1f7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8d2013f710 0x7f8d201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39044/0 (socket says 172.21.15.32:39044) 2024-03-25T18:44:07.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.891+0000 7f8d1f7fe640 1 -- 172.21.15.32:0/2805169748 learned_addr learned my addr 172.21.15.32:0/2805169748 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:07.893 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:20.210 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.901 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:20.218 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.905 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:20.233 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.910 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:20.238 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.918+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2686254209 0 0) 0x7f8d200b28a0 con 0x7f8d201430c0 2024-03-25T18:44:07.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.918+0000 7f8d1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:07.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.918+0000 7f8d1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:07.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.918+0000 7f8d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:07.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.918+0000 7f8d1effd640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:44:07.920 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.918+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d08003520 con 0x7f8d201430c0 2024-03-25T18:44:07.942 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:20.270 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3570079285 0 0) 0x7f8d200b2a30 con 0x7f8d200a98f0 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d200b28a0 con 0x7f8d200a98f0 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775728794 0 0) 0x7f8d20142520 con 0x7f8d2013f710 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:07.943 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:44:07.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.943+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d200b2a30 con 0x7f8d2013f710 2024-03-25T18:44:07.944 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:20.261 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125182936 0 0) 0x7f8d08003520 con 0x7f8d201430c0 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 >> v1:172.21.15.32:6790/0 conn(0x7f8d200a98f0 legacy=0x7f8d200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 >> v1:172.21.15.32:6789/0 conn(0x7f8d2013f710 legacy=0x7f8d201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 10 monclient: _finish_hunting 0 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 1 monclient: found mon.noname-c 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d200b44c0 con 0x7f8d201430c0 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 10 monclient: _finish_auth 0 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 10 monclient: _check_auth_tickets 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8d18001a10 con 0x7f8d201430c0 2024-03-25T18:44:07.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d263af640 5 monclient: authenticate success, global_id 4429 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d263af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d1effd640 10 monclient: dump: 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:07.954 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:07.955 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:07.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:07.952+0000 7f8d263af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:08.007 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d18001b70 con 0x7f8d201430c0 2024-03-25T18:44:08.007 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:08.007 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d1effd640 1 -- 172.21.15.32:0/2805169748 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8d18001f80 con 0x7f8d201430c0 2024-03-25T18:44:08.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d263af640 10 monclient: get_monmap_and_config success 2024-03-25T18:44:08.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d263af640 4 set_mon_vals no callback set 2024-03-25T18:44:08.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:08.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d1effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:08.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d1effd640 10 monclient: dump: 2024-03-25T18:44:08.008 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:08.009 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:08.009 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:08.009 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:08.009 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:08.009 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:08.010 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:08.010 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:08.010 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:08.010 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:08.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d263af640 10 monclient: shutdown 2024-03-25T18:44:08.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d263af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:08.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d263af640 1 -- 172.21.15.32:0/2805169748 >> v1:172.21.15.136:6789/0 conn(0x7f8d201430c0 legacy=0x7f8d2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:08.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.007+0000 7f8d263af640 1 -- 172.21.15.32:0/2805169748 shutdown_connections 2024-03-25T18:44:08.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.008+0000 7f8d263af640 1 -- 172.21.15.32:0/2805169748 >> 172.21.15.32:0/2805169748 conn(0x7f8d200b7ea0 msgr2=0x7f8d200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:08.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.008+0000 7f8d263af640 1 -- 172.21.15.32:0/2805169748 shutdown_connections 2024-03-25T18:44:08.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.008+0000 7f8d263af640 1 -- 172.21.15.32:0/2805169748 wait complete. 2024-03-25T18:44:08.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.008+0000 7f8d263af640 10 monclient: build_initial_monmap 2024-03-25T18:44:08.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.008+0000 7f8d263af640 10 monclient: monmap: 2024-03-25T18:44:08.015 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:08.016 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.009+0000 7f8d263af640 1 librados: starting msgr at 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.009+0000 7f8d263af640 1 librados: starting objecter 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.009+0000 7f8d263af640 1 Processor -- start 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.009+0000 7f8d263af640 1 -- start start 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.009+0000 7f8d263af640 1 librados: setting wanted keys 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.009+0000 7f8d263af640 1 librados: calling monclient init 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.009+0000 7f8d263af640 10 monclient: init 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 10 monclient(hunting): picked mon.noname-c con 0x7f8d200a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 10 monclient(hunting): picked mon.noname-a con 0x7f8d2013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:08.017 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 10 monclient(hunting): picked mon.noname-b con 0x7f8d201430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:08.018 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d201db170 con 0x7f8d2013f710 2024-03-25T18:44:08.018 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d201dac50 con 0x7f8d201430c0 2024-03-25T18:44:08.018 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d201dade0 con 0x7f8d200a98f0 2024-03-25T18:44:08.018 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:08.018 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.010+0000 7f8d263af640 10 monclient(hunting): authenticate will time out at 6354.198674s 2024-03-25T18:44:08.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.024+0000 7f8d1f7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f8d2013f710 0x7f8d201d7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:39112/0 (socket says 172.21.15.32:39112) 2024-03-25T18:44:08.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.024+0000 7f8d1f7fe640 1 -- 172.21.15.32:0/1107777718 learned_addr learned my addr 172.21.15.32:0/1107777718 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:08.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.025+0000 7f8d1cff9640 10 client.?.objecter ms_handle_connect 0x7f8d2013f710 2024-03-25T18:44:08.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.025+0000 7f8d1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:08.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.029+0000 7f8d1cff9640 10 client.?.objecter ms_handle_connect 0x7f8d201430c0 2024-03-25T18:44:08.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.029+0000 7f8d1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:08.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.040+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779454825 0 0) 0x7f8d201dac50 con 0x7f8d201430c0 2024-03-25T18:44:08.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.040+0000 7f8d1cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:08.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.040+0000 7f8d1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:08.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.040+0000 7f8d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:08.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.040+0000 7f8d1cff9640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:44:08.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.040+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d0c004eb0 con 0x7f8d201430c0 2024-03-25T18:44:08.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.043+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2044098683 0 0) 0x7f8d201db170 con 0x7f8d2013f710 2024-03-25T18:44:08.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.043+0000 7f8d1cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:08.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.043+0000 7f8d1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:08.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.043+0000 7f8d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:08.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.043+0000 7f8d1cff9640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:44:08.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.043+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d201dac50 con 0x7f8d2013f710 2024-03-25T18:44:08.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.046+0000 7f8d1cff9640 10 client.?.objecter ms_handle_connect 0x7f8d200a98f0 2024-03-25T18:44:08.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.046+0000 7f8d1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:08.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.058+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1749133188 0 0) 0x7f8d0c004eb0 con 0x7f8d201430c0 2024-03-25T18:44:08.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.058+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d201db170 con 0x7f8d201430c0 2024-03-25T18:44:08.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.058+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8d18001e80 con 0x7f8d201430c0 2024-03-25T18:44:08.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.058+0000 7f8d1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.058+0000 7f8d1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.058+0000 7f8d1cff9640 10 monclient(hunting): dump: 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:08.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.058+0000 7f8d1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:44:08.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.062+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034824146 0 0) 0x7f8d201dade0 con 0x7f8d200a98f0 2024-03-25T18:44:08.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.062+0000 7f8d1cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:08.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.062+0000 7f8d1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:08.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.062+0000 7f8d1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:08.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.062+0000 7f8d1cff9640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:44:08.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.062+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d0c004eb0 con 0x7f8d200a98f0 2024-03-25T18:44:08.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.072+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1158806398 0 0) 0x7f8d201db170 con 0x7f8d201430c0 2024-03-25T18:44:08.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.072+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 >> v1:172.21.15.32:6790/0 conn(0x7f8d200a98f0 legacy=0x7f8d201394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:08.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.072+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 >> v1:172.21.15.32:6789/0 conn(0x7f8d2013f710 legacy=0x7f8d201d7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:08.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.072+0000 7f8d1cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:44:08.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.072+0000 7f8d1cff9640 1 monclient: found mon.b 2024-03-25T18:44:08.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.072+0000 7f8d1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:08.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.072+0000 7f8d1cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:08.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2013c170 con 0x7f8d201430c0 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d1cff9640 10 monclient: _finish_auth 0 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d1cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 5 monclient: authenticate success, global_id 4432 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 10 monclient: _renew_subs 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2013c420 con 0x7f8d201430c0 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 10 client.4432.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 10 monclient: _renew_subs 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d20133860 con 0x7f8d201430c0 2024-03-25T18:44:08.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.073+0000 7f8d263af640 1 librados: init done 2024-03-25T18:44:08.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.074+0000 7f8d263af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:08.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.074+0000 7f8d263af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:08.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.074+0000 7f8d263af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:08.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.074+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d201daf70 con 0x7f8d201430c0 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.346+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d18002040 con 0x7f8d201430c0 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.346+0000 7f8d1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.346+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8d18003dc0 con 0x7f8d201430c0 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.346+0000 7f8d1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.346+0000 7f8d1cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.346+0000 7f8d1cff9640 10 monclient: dump: 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:08.347 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.346+0000 7f8d1e7fc640 4 set_mon_vals no callback set 2024-03-25T18:44:08.348 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.346+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f8d18019d90 con 0x7f8d201430c0 2024-03-25T18:44:08.349 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.347+0000 7f8d1ffff640 10 monclient: get_auth_request con 0x7f8d0c06dd60 auth_method 0 2024-03-25T18:44:08.349 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.347+0000 7f8d1cff9640 10 client.4432.objecter ms_handle_connect 0x7f8d0c06dd60 2024-03-25T18:44:08.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.352+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f8d180538b0 con 0x7f8d201430c0 2024-03-25T18:44:08.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.352+0000 7f8d1cff9640 10 client.4432.objecter ms_dispatch 0x7f8d20000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:44:08.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.352+0000 7f8d1cff9640 3 client.4432.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:44:08.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.352+0000 7f8d1cff9640 3 client.4432.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:44:08.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.352+0000 7f8d1cff9640 20 client.4432.objecter dump_active .. 0 homeless 2024-03-25T18:44:08.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.353+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d1804f3c0 con 0x7f8d201430c0 2024-03-25T18:44:08.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.353+0000 7f8d1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:08.353 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.353+0000 7f8d1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:44:08.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.616+0000 7f8d263af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:08.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.616+0000 7f8d263af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:08.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.616+0000 7f8d263af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:08.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.616+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d200627b0 con 0x7f8d201430c0 2024-03-25T18:44:08.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.641+0000 7f8d1cff9640 1 -- 172.21.15.32:0/1107777718 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f8d180512f0 con 0x7f8d201430c0 2024-03-25T18:44:08.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.641+0000 7f8d1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:08.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.641+0000 7f8d1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:44:08.641 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:44:08.641 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:44:08.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 10 librados: watch_flush enter 2024-03-25T18:44:08.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 10 librados: watch_flush exit 2024-03-25T18:44:08.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 20 client.4432.objecter shutdown clearing up homeless session... 2024-03-25T18:44:08.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 10 client.4432.objecter successfully canceled tick 2024-03-25T18:44:08.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 >> v1:172.21.15.136:6800/1141502808 conn(0x7f8d0c06dd60 legacy=0x7f8d0c070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:08.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 10 monclient: shutdown 2024-03-25T18:44:08.644 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:08.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 >> v1:172.21.15.136:6789/0 conn(0x7f8d201430c0 legacy=0x7f8d20132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:08.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 shutdown_connections 2024-03-25T18:44:08.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 >> 172.21.15.32:0/1107777718 conn(0x7f8d200b7ea0 msgr2=0x7f8d2014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:08.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 shutdown_connections 2024-03-25T18:44:08.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.643+0000 7f8d263af640 1 -- 172.21.15.32:0/1107777718 wait complete. 2024-03-25T18:44:08.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:08.644+0000 7f8d263af640 1 librados: shutdown 2024-03-25T18:44:08.655 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:44:15.461 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:44:15.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.576+0000 7fa6a1a3e640 10 monclient: get_monmap_and_config 2024-03-25T18:44:15.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.576+0000 7fa6a1a3e640 10 monclient: build_initial_monmap 2024-03-25T18:44:15.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.576+0000 7fa6a1a3e640 10 monclient: monmap: 2024-03-25T18:44:15.581 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:15.581 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:15.581 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:44:15.578185+0000 2024-03-25T18:44:15.581 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:44:15.578185+0000 2024-03-25T18:44:15.581 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:15.581 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:15.582 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:15.582 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:44:15.582 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:44:15.582 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:15.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.577+0000 7fa6a1a3e640 1 Processor -- start 2024-03-25T18:44:15.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.577+0000 7fa6a1a3e640 1 -- start start 2024-03-25T18:44:15.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.577+0000 7fa6a1a3e640 10 monclient: init 2024-03-25T18:44:15.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:15.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:44:15.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 10 monclient(hunting): picked mon.noname-b con 0x7fa69c0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:15.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 10 monclient(hunting): picked mon.noname-a con 0x7fa69c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:15.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 10 monclient(hunting): picked mon.noname-c con 0x7fa69c1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:15.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c142520 con 0x7fa69c13f710 2024-03-25T18:44:15.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c0b28a0 con 0x7fa69c1430c0 2024-03-25T18:44:15.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c0b2a30 con 0x7fa69c0a98f0 2024-03-25T18:44:15.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:15.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.578+0000 7fa6a1a3e640 10 monclient(hunting): authenticate will time out at 6361.768068s 2024-03-25T18:44:15.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.594+0000 7fa69b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa69c1430c0 0x7fa69c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43912/0 (socket says 172.21.15.32:43912) 2024-03-25T18:44:15.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.594+0000 7fa69b7fe640 1 -- 172.21.15.32:0/1347453650 learned_addr learned my addr 172.21.15.32:0/1347453650 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:15.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.606+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830623193 0 0) 0x7fa69c0b2a30 con 0x7fa69c0a98f0 2024-03-25T18:44:15.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.606+0000 7fa69a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:44:15.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.606+0000 7fa69a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:15.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.606+0000 7fa69a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:15.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.606+0000 7fa69a7fc640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:44:15.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.606+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa688003520 con 0x7fa69c0a98f0 2024-03-25T18:44:15.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.609+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1553832879 0 0) 0x7fa69c0b28a0 con 0x7fa69c1430c0 2024-03-25T18:44:15.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.609+0000 7fa69a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:44:15.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.609+0000 7fa69a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:15.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.609+0000 7fa69a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:15.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.609+0000 7fa69a7fc640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:44:15.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.609+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69c0b2a30 con 0x7fa69c1430c0 2024-03-25T18:44:15.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138549735 0 0) 0x7fa69c142520 con 0x7fa69c13f710 2024-03-25T18:44:15.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:44:15.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:15.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69c0b28a0 con 0x7fa69c13f710 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2727007137 0 0) 0x7fa688003520 con 0x7fa69c0a98f0 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 >> v1:172.21.15.136:6789/0 conn(0x7fa69c1430c0 legacy=0x7fa69c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 >> v1:172.21.15.32:6789/0 conn(0x7fa69c13f710 legacy=0x7fa69c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 1 monclient: found mon.noname-b 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c0b44c0 con 0x7fa69c0a98f0 2024-03-25T18:44:15.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa68c001690 con 0x7fa69c0a98f0 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa6a1a3e640 5 monclient: authenticate success, global_id 4439 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa6a1a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa69a7fc640 10 monclient: dump: 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.616 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:15.617 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:15.617 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:15.617 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:15.617 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:15.617 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:15.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.612+0000 7fa6a1a3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:15.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.646+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa68c0017f0 con 0x7fa69c0a98f0 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.646+0000 7fa69a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.646+0000 7fa69a7fc640 1 -- 172.21.15.32:0/1347453650 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa68c001c00 con 0x7fa69c0a98f0 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.646+0000 7fa69a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.646+0000 7fa69a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.646+0000 7fa69a7fc640 10 monclient: dump: 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:15.648 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:15.649 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:15.649 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:15.649 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:15.649 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:15.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.646+0000 7fa6a1a3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:44:15.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.646+0000 7fa6a1a3e640 4 set_mon_vals no callback set 2024-03-25T18:44:15.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.647+0000 7fa6a1a3e640 10 monclient: shutdown 2024-03-25T18:44:15.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.647+0000 7fa6a1a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:15.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.647+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/1347453650 >> v1:172.21.15.32:6790/0 conn(0x7fa69c0a98f0 legacy=0x7fa69c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:15.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.647+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/1347453650 shutdown_connections 2024-03-25T18:44:15.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.647+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/1347453650 >> 172.21.15.32:0/1347453650 conn(0x7fa69c0b7ea0 msgr2=0x7fa69c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.647+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/1347453650 shutdown_connections 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.647+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/1347453650 wait complete. 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 10 monclient: build_initial_monmap 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 10 monclient: monmap: 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:15.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 1 librados: starting msgr at 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 1 librados: starting objecter 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 1 Processor -- start 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 1 -- start start 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 1 librados: setting wanted keys 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 1 librados: calling monclient init 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.648+0000 7fa6a1a3e640 10 monclient: init 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.649+0000 7fa6a1a3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.649+0000 7fa6a1a3e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.649+0000 7fa6a1a3e640 10 monclient(hunting): picked mon.noname-a con 0x7fa69c0a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.649+0000 7fa6a1a3e640 10 monclient(hunting): picked mon.noname-c con 0x7fa69c13f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.649+0000 7fa6a1a3e640 10 monclient(hunting): picked mon.noname-b con 0x7fa69c1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.649+0000 7fa6a1a3e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c137ab0 con 0x7fa69c0a98f0 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.650+0000 7fa6a1a3e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c138bd0 con 0x7fa69c1430c0 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.650+0000 7fa6a1a3e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa69c138e40 con 0x7fa69c13f710 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.650+0000 7fa6a1a3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:15.653 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.650+0000 7fa6a1a3e640 10 monclient(hunting): authenticate will time out at 6361.839613s 2024-03-25T18:44:15.656 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.655+0000 7fa693fff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fa69c13f710 0x7fa69c1df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:37610/0 (socket says 172.21.15.32:37610) 2024-03-25T18:44:15.656 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.655+0000 7fa693fff640 1 -- 172.21.15.32:0/3903033499 learned_addr learned my addr 172.21.15.32:0/3903033499 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:15.657 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.656+0000 7fa698ff9640 10 client.?.objecter ms_handle_connect 0x7fa69c13f710 2024-03-25T18:44:15.657 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.656+0000 7fa698ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:15.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.670+0000 7fa698ff9640 10 client.?.objecter ms_handle_connect 0x7fa69c1430c0 2024-03-25T18:44:15.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.670+0000 7fa698ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:15.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.671+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324324155 0 0) 0x7fa69c138e40 con 0x7fa69c13f710 2024-03-25T18:44:15.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.671+0000 7fa698ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:15.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.671+0000 7fa698ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:15.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.671+0000 7fa698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:15.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.671+0000 7fa698ff9640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:44:15.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.671+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa67c004eb0 con 0x7fa69c13f710 2024-03-25T18:44:15.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.673+0000 7fa698ff9640 10 client.?.objecter ms_handle_connect 0x7fa69c0a98f0 2024-03-25T18:44:15.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.673+0000 7fa698ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:15.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316752619 0 0) 0x7fa69c137ab0 con 0x7fa69c0a98f0 2024-03-25T18:44:15.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:15.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:15.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:15.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:44:15.680 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69c138e40 con 0x7fa69c0a98f0 2024-03-25T18:44:15.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 353295676 0 0) 0x7fa69c138bd0 con 0x7fa69c1430c0 2024-03-25T18:44:15.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:15.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:15.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:15.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:44:15.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.679+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa69c137ab0 con 0x7fa69c1430c0 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.690+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2940323420 0 0) 0x7fa69c138e40 con 0x7fa69c0a98f0 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.690+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa69c138bd0 con 0x7fa69c0a98f0 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.690+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa68c001b00 con 0x7fa69c0a98f0 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.690+0000 7fa698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.690+0000 7fa698ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.690+0000 7fa698ff9640 10 monclient(hunting): dump: 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:15.692 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:15.693 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:15.693 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:15.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.690+0000 7fa698ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:44:15.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.693+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1756770566 0 0) 0x7fa67c004eb0 con 0x7fa69c13f710 2024-03-25T18:44:15.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.694+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa69c138e40 con 0x7fa69c13f710 2024-03-25T18:44:15.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.694+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa684001a10 con 0x7fa69c13f710 2024-03-25T18:44:15.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.694+0000 7fa698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:15.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.694+0000 7fa698ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:44:15.695 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.694+0000 7fa698ff9640 10 monclient(hunting): dump: 2024-03-25T18:44:15.695 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:15.696 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:15.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.696+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 987004954 0 0) 0x7fa69c137ab0 con 0x7fa69c1430c0 2024-03-25T18:44:15.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.696+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa67c004eb0 con 0x7fa69c1430c0 2024-03-25T18:44:15.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.696+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa694001bd0 con 0x7fa69c1430c0 2024-03-25T18:44:15.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.696+0000 7fa698ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:15.697 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.696+0000 7fa698ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.696+0000 7fa698ff9640 10 monclient(hunting): dump: 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:15.698 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:15.712 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3109517236 0 0) 0x7fa67c004eb0 con 0x7fa69c1430c0 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 >> v1:172.21.15.32:6790/0 conn(0x7fa69c13f710 legacy=0x7fa69c1df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 >> v1:172.21.15.32:6789/0 conn(0x7fa69c0a98f0 legacy=0x7fa69c137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 1 monclient: found mon.b 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa69c139120 con 0x7fa69c1430c0 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 10 monclient: _finish_auth 0 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa698ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 10 monclient: _renew_subs 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:15.713 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa69c139400 con 0x7fa69c1430c0 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 10 client.4438.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 10 monclient: _renew_subs 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa69c0b7880 con 0x7fa69c1430c0 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.710+0000 7fa6a1a3e640 1 librados: init done 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.711+0000 7fa6a1a3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.711+0000 7fa6a1a3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.711+0000 7fa6a1a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:15.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:15.711+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa69c137810 con 0x7fa69c1430c0 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.007+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa694002000 con 0x7fa69c1430c0 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.007+0000 7fa698ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.007+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa694004d00 con 0x7fa69c1430c0 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.007+0000 7fa698ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.007+0000 7fa698ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.007+0000 7fa698ff9640 10 monclient: dump: 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:16.009 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:16.010 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:16.010 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:16.010 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:16.010 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:16.010 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:16.010 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:16.010 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:16.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.007+0000 7fa699ffb640 4 set_mon_vals no callback set 2024-03-25T18:44:16.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.008+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fa69401acd0 con 0x7fa69c1430c0 2024-03-25T18:44:16.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.009+0000 7fa69affd640 10 monclient: get_auth_request con 0x7fa67c06e160 auth_method 0 2024-03-25T18:44:16.011 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.009+0000 7fa698ff9640 10 client.4438.objecter ms_handle_connect 0x7fa67c06e160 2024-03-25T18:44:16.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.013+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fa694054980 con 0x7fa69c1430c0 2024-03-25T18:44:16.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.013+0000 7fa698ff9640 10 client.4438.objecter ms_dispatch 0x7fa69c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:44:16.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.013+0000 7fa698ff9640 3 client.4438.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:44:16.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.013+0000 7fa698ff9640 3 client.4438.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:44:16.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.013+0000 7fa698ff9640 20 client.4438.objecter dump_active .. 0 homeless 2024-03-25T18:44:16.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.014+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa6940502c0 con 0x7fa69c1430c0 2024-03-25T18:44:16.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.014+0000 7fa698ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:16.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.014+0000 7fa698ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:44:16.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.268+0000 7fa6a1a3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:16.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.268+0000 7fa6a1a3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:16.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.268+0000 7fa6a1a3e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:16.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.268+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa69c0627b0 con 0x7fa69c1430c0 2024-03-25T18:44:16.318 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.317+0000 7fa698ff9640 1 -- 172.21.15.32:0/3903033499 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fa6940523c0 con 0x7fa69c1430c0 2024-03-25T18:44:16.319 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.317+0000 7fa698ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:16.319 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.317+0000 7fa698ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:44:16.319 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:44:16.319 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:44:16.321 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.319+0000 7fa6a1a3e640 10 librados: watch_flush enter 2024-03-25T18:44:16.321 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 10 librados: watch_flush exit 2024-03-25T18:44:16.321 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 20 client.4438.objecter shutdown clearing up homeless session... 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 10 client.4438.objecter successfully canceled tick 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 >> v1:172.21.15.136:6800/1141502808 conn(0x7fa67c06e160 legacy=0x7fa67c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 10 monclient: shutdown 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 >> v1:172.21.15.136:6789/0 conn(0x7fa69c1430c0 legacy=0x7fa69c1e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 shutdown_connections 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 >> 172.21.15.32:0/3903033499 conn(0x7fa69c0b7ea0 msgr2=0x7fa69c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 shutdown_connections 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 1 -- 172.21.15.32:0/3903033499 wait complete. 2024-03-25T18:44:16.322 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:16.320+0000 7fa6a1a3e640 1 librados: shutdown 2024-03-25T18:44:16.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:44:23.140 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:44:23.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.253+0000 7f5392bac640 10 monclient: get_monmap_and_config 2024-03-25T18:44:23.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.253+0000 7f5392bac640 10 monclient: build_initial_monmap 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.253+0000 7f5392bac640 10 monclient: monmap: 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:44:23.254701+0000 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:44:23.254701+0000 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:44:23.255 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:23.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.254+0000 7f5392bac640 1 Processor -- start 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.254+0000 7f5392bac640 1 -- start start 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.254+0000 7f5392bac640 10 monclient: init 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.254+0000 7f5392bac640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.254+0000 7f5392bac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.254+0000 7f5392bac640 10 monclient(hunting): picked mon.noname-c con 0x7f538c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.254+0000 7f5392bac640 10 monclient(hunting): picked mon.noname-a con 0x7f538c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.255+0000 7f5392bac640 10 monclient(hunting): picked mon.noname-b con 0x7f538c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.255+0000 7f5392bac640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c142520 con 0x7f538c13f710 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.255+0000 7f5392bac640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c0b28a0 con 0x7f538c0a98f0 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.255+0000 7f5392bac640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c0b2a30 con 0x7f538c1430c0 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.255+0000 7f5392bac640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:23.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.255+0000 7f5392bac640 10 monclient(hunting): authenticate will time out at 6369.444702s 2024-03-25T18:44:23.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.265+0000 7f5383fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f538c13f710 0x7f538c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46378/0 (socket says 172.21.15.32:46378) 2024-03-25T18:44:23.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.265+0000 7f5383fff640 1 -- 172.21.15.32:0/1913739488 learned_addr learned my addr 172.21.15.32:0/1913739488 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:23.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.277+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042763285 0 0) 0x7f538c142520 con 0x7f538c13f710 2024-03-25T18:44:23.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.277+0000 7f53837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:44:23.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.277+0000 7f53837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:23.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.277+0000 7f53837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:23.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.277+0000 7f53837fe640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:44:23.278 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.277+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5374003520 con 0x7f538c13f710 2024-03-25T18:44:23.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.294+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3420735689 0 0) 0x7f538c0b2a30 con 0x7f538c1430c0 2024-03-25T18:44:23.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.294+0000 7f53837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:44:23.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.294+0000 7f53837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:23.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.294+0000 7f53837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:23.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.295+0000 7f53837fe640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:44:23.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.295+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f538c142520 con 0x7f538c1430c0 2024-03-25T18:44:23.299 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.298+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1793206267 0 0) 0x7f538c0b28a0 con 0x7f538c0a98f0 2024-03-25T18:44:23.299 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.298+0000 7f53837fe640 10 monclient(hunting): _negotiate 2024-03-25T18:44:23.299 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.298+0000 7f53837fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:23.299 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.298+0000 7f53837fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:23.299 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.298+0000 7f53837fe640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:44:23.299 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.298+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f538c0b2a30 con 0x7f538c0a98f0 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2431543248 0 0) 0x7f5374003520 con 0x7f538c13f710 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 >> v1:172.21.15.32:6790/0 conn(0x7f538c1430c0 legacy=0x7f538c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 >> v1:172.21.15.136:6789/0 conn(0x7f538c0a98f0 legacy=0x7f538c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 10 monclient: _finish_hunting 0 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 1 monclient: found mon.noname-a 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c0b44c0 con 0x7f538c13f710 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 10 monclient: _finish_auth 0 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 10 monclient: _check_auth_tickets 2024-03-25T18:44:23.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f537c001690 con 0x7f538c13f710 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f53837fe640 10 monclient: dump: 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:23.304 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:23.305 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:23.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f5392bac640 5 monclient: authenticate success, global_id 4458 2024-03-25T18:44:23.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.301+0000 7f5392bac640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:23.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.323+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f537c0017f0 con 0x7f538c13f710 2024-03-25T18:44:23.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.323+0000 7f53837fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:23.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.323+0000 7f53837fe640 1 -- 172.21.15.32:0/1913739488 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f537c001c00 con 0x7f538c13f710 2024-03-25T18:44:23.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.323+0000 7f5392bac640 10 monclient: get_monmap_and_config success 2024-03-25T18:44:23.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.323+0000 7f5392bac640 4 set_mon_vals no callback set 2024-03-25T18:44:23.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.323+0000 7f53837fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:23.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.323+0000 7f53837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:44:23.324 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.323+0000 7f53837fe640 10 monclient: dump: 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:23.325 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:23.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.324+0000 7f5392bac640 10 monclient: shutdown 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.324+0000 7f5392bac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.324+0000 7f5392bac640 1 -- 172.21.15.32:0/1913739488 >> v1:172.21.15.32:6789/0 conn(0x7f538c13f710 legacy=0x7f538c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.324+0000 7f5392bac640 1 -- 172.21.15.32:0/1913739488 shutdown_connections 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.324+0000 7f5392bac640 1 -- 172.21.15.32:0/1913739488 >> 172.21.15.32:0/1913739488 conn(0x7f538c0b7ea0 msgr2=0x7f538c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.324+0000 7f5392bac640 1 -- 172.21.15.32:0/1913739488 shutdown_connections 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.324+0000 7f5392bac640 1 -- 172.21.15.32:0/1913739488 wait complete. 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 10 monclient: build_initial_monmap 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 10 monclient: monmap: 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:44:23.327 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 1 librados: starting msgr at 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 1 librados: starting objecter 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 1 Processor -- start 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 1 -- start start 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 1 librados: setting wanted keys 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 1 librados: calling monclient init 2024-03-25T18:44:23.328 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.325+0000 7f5392bac640 10 monclient: init 2024-03-25T18:44:23.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.326+0000 7f5392bac640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:23.329 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.326+0000 7f5392bac640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:44:23.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.326+0000 7f5392bac640 10 monclient(hunting): picked mon.noname-b con 0x7f538c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:23.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.326+0000 7f5392bac640 10 monclient(hunting): picked mon.noname-a con 0x7f538c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:23.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.326+0000 7f5392bac640 10 monclient(hunting): picked mon.noname-c con 0x7f538c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:23.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.326+0000 7f5392bac640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c135550 con 0x7f538c13f710 2024-03-25T18:44:23.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.327+0000 7f5392bac640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c136b50 con 0x7f538c0a98f0 2024-03-25T18:44:23.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.327+0000 7f5392bac640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f538c136e30 con 0x7f538c1430c0 2024-03-25T18:44:23.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.327+0000 7f5392bac640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:23.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.327+0000 7f5392bac640 10 monclient(hunting): authenticate will time out at 6369.516572s 2024-03-25T18:44:23.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.330+0000 7f5383fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f538c13f710 0x7f538c1375a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46466/0 (socket says 172.21.15.32:46466) 2024-03-25T18:44:23.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.330+0000 7f5383fff640 1 -- 172.21.15.32:0/2444610924 learned_addr learned my addr 172.21.15.32:0/2444610924 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:23.336 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.335+0000 7f53817fa640 10 client.?.objecter ms_handle_connect 0x7f538c13f710 2024-03-25T18:44:23.336 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.335+0000 7f53817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:23.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.350+0000 7f53817fa640 10 client.?.objecter ms_handle_connect 0x7f538c0a98f0 2024-03-25T18:44:23.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.350+0000 7f53817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:23.355 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.354+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111172184 0 0) 0x7f538c136b50 con 0x7f538c0a98f0 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.354+0000 7f53817fa640 10 monclient(hunting): _negotiate 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.354+0000 7f53817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.354+0000 7f53817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.354+0000 7f53817fa640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.354+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5378004eb0 con 0x7f538c0a98f0 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.355+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1990562483 0 0) 0x7f538c135550 con 0x7f538c13f710 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.355+0000 7f53817fa640 10 monclient(hunting): _negotiate 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.355+0000 7f53817fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.355+0000 7f53817fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.355+0000 7f53817fa640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:44:23.356 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.355+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f538c136b50 con 0x7f538c13f710 2024-03-25T18:44:23.362 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.361+0000 7f53817fa640 10 client.?.objecter ms_handle_connect 0x7f538c1430c0 2024-03-25T18:44:23.362 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.361+0000 7f53817fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.363+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1582155550 0 0) 0x7f538c136b50 con 0x7f538c13f710 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.363+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f538c135550 con 0x7f538c13f710 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.363+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f537c001b80 con 0x7f538c13f710 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.363+0000 7f53817fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.363+0000 7f53817fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.363+0000 7f53817fa640 10 monclient(hunting): dump: 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:23.365 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:23.366 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:23.366 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:23.366 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:23.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.363+0000 7f53817fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:44:23.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3988036582 0 0) 0x7f538c135550 con 0x7f538c13f710 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 >> v1:172.21.15.32:6790/0 conn(0x7f538c1430c0 legacy=0x7f538c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 >> v1:172.21.15.136:6789/0 conn(0x7f538c0a98f0 legacy=0x7f538c13b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 10 monclient: _finish_hunting 0 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 1 monclient: found mon.a 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f538c1357f0 con 0x7f538c13f710 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 10 monclient: _finish_auth 0 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 10 monclient: _check_auth_tickets 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f53817fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 5 monclient: authenticate success, global_id 4461 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 10 monclient: _renew_subs 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:23.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f538c137cb0 con 0x7f538c13f710 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 10 client.4461.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 10 monclient: _renew_subs 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f538c1352e0 con 0x7f538c13f710 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.373+0000 7f5392bac640 1 librados: init done 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.374+0000 7f5392bac640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.374+0000 7f5392bac640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.374+0000 7f5392bac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:23.377 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.374+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f538c0627b0 con 0x7f538c13f710 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.685+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f537c0011f0 con 0x7f538c13f710 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.685+0000 7f53817fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.685+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f537c004cb0 con 0x7f538c13f710 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.685+0000 7f53817fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.685+0000 7f53817fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.685+0000 7f53817fa640 10 monclient: dump: 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:23.687 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:23.688 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:23.688 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:23.688 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:23.688 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:23.688 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:23.688 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:23.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.685+0000 7f5382ffd640 4 set_mon_vals no callback set 2024-03-25T18:44:23.689 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.686+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f537c01ac80 con 0x7f538c13f710 2024-03-25T18:44:23.689 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.687+0000 7f5390948640 10 monclient: get_auth_request con 0x7f537806dd00 auth_method 0 2024-03-25T18:44:23.689 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.687+0000 7f53817fa640 10 client.4461.objecter ms_handle_connect 0x7f537806dd00 2024-03-25T18:44:23.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.692+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f537c0548e0 con 0x7f538c13f710 2024-03-25T18:44:23.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.692+0000 7f53817fa640 10 client.4461.objecter ms_dispatch 0x7f538c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:44:23.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.692+0000 7f53817fa640 3 client.4461.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:44:23.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.692+0000 7f53817fa640 3 client.4461.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:44:23.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.692+0000 7f53817fa640 20 client.4461.objecter dump_active .. 0 homeless 2024-03-25T18:44:23.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.692+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f537c050220 con 0x7f538c13f710 2024-03-25T18:44:23.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.692+0000 7f53817fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:23.693 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.692+0000 7f53817fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:44:23.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.943+0000 7f5392bac640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:23.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.943+0000 7f5392bac640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:23.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.943+0000 7f5392bac640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:44:23.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.943+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f538c13e210 con 0x7f538c13f710 2024-03-25T18:44:23.971 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.970+0000 7f53817fa640 1 -- 172.21.15.32:0/2444610924 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f537c052320 con 0x7f538c13f710 2024-03-25T18:44:23.971 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.970+0000 7f53817fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:23.971 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.970+0000 7f53817fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:44:23.972 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:44:23.972 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:44:23.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.972+0000 7f5392bac640 10 librados: watch_flush enter 2024-03-25T18:44:23.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.972+0000 7f5392bac640 10 librados: watch_flush exit 2024-03-25T18:44:23.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.972+0000 7f5392bac640 20 client.4461.objecter shutdown clearing up homeless session... 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.972+0000 7f5392bac640 10 client.4461.objecter successfully canceled tick 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.972+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 >> v1:172.21.15.136:6800/1141502808 conn(0x7f537806dd00 legacy=0x7f53780701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.973+0000 7f5392bac640 10 monclient: shutdown 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.973+0000 7f5392bac640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.973+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 >> v1:172.21.15.32:6789/0 conn(0x7f538c13f710 legacy=0x7f538c1375a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.973+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 shutdown_connections 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.973+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 >> 172.21.15.32:0/2444610924 conn(0x7f538c0b7ea0 msgr2=0x7f538c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.973+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 shutdown_connections 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.973+0000 7f5392bac640 1 -- 172.21.15.32:0/2444610924 wait complete. 2024-03-25T18:44:23.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:23.973+0000 7f5392bac640 1 librados: shutdown 2024-03-25T18:44:23.987 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:44:27.920 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:40.247 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:27.927 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:40.255 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:27.951 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:05:40.278 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:27.982 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:40.299 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:27.982 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:40.310 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:27.986 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:40.314 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:27.988 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:40.305 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:28.029 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:05:40.346 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:28.034 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:05:40.362 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:30.794 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:44:30.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.912+0000 7f507420f640 10 monclient: get_monmap_and_config 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.912+0000 7f507420f640 10 monclient: build_initial_monmap 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.912+0000 7f507420f640 10 monclient: monmap: 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:44:30.914078+0000 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:44:30.914078+0000 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:44:30.914 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.913+0000 7f507420f640 1 Processor -- start 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.913+0000 7f507420f640 1 -- start start 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.913+0000 7f507420f640 10 monclient: init 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 10 monclient(hunting): picked mon.noname-a con 0x7f506c0a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 10 monclient(hunting): picked mon.noname-b con 0x7f506c13f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 10 monclient(hunting): picked mon.noname-c con 0x7f506c1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c142520 con 0x7f506c0a98f0 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c0b28a0 con 0x7f506c1430c0 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c0b2a30 con 0x7f506c13f710 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:30.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.914+0000 7f507420f640 10 monclient(hunting): authenticate will time out at 6377.104057s 2024-03-25T18:44:30.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.923+0000 7f50727ac640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f506c1430c0 0x7f506c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:57734/0 (socket says 172.21.15.32:57734) 2024-03-25T18:44:30.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.923+0000 7f50727ac640 1 -- 172.21.15.32:0/3285911248 learned_addr learned my addr 172.21.15.32:0/3285911248 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:30.938 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.937+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34940449 0 0) 0x7f506c142520 con 0x7f506c0a98f0 2024-03-25T18:44:30.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.937+0000 7f5070fa9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:30.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.937+0000 7f5070fa9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:30.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.937+0000 7f5070fa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:30.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.937+0000 7f5070fa9640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:44:30.939 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.937+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5058003520 con 0x7f506c0a98f0 2024-03-25T18:44:30.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.943+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156150909 0 0) 0x7f506c0b28a0 con 0x7f506c1430c0 2024-03-25T18:44:30.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.943+0000 7f5070fa9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:30.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.943+0000 7f5070fa9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:30.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.943+0000 7f5070fa9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:30.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.943+0000 7f5070fa9640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:44:30.944 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.943+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506c142520 con 0x7f506c1430c0 2024-03-25T18:44:30.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1105256114 0 0) 0x7f5058003520 con 0x7f506c0a98f0 2024-03-25T18:44:30.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 >> v1:172.21.15.32:6790/0 conn(0x7f506c13f710 legacy=0x7f506c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:30.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 >> v1:172.21.15.136:6789/0 conn(0x7f506c1430c0 legacy=0x7f506c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:30.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 10 monclient: _finish_hunting 0 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 1 monclient: found mon.noname-a 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c0b44c0 con 0x7f506c0a98f0 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 10 monclient: _finish_auth 0 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 10 monclient: _check_auth_tickets 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5068001690 con 0x7f506c0a98f0 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f5070fa9640 10 monclient: dump: 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:30.948 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:30.949 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:30.949 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:30.949 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:30.949 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:30.949 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:30.949 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:30.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f507420f640 5 monclient: authenticate success, global_id 4464 2024-03-25T18:44:30.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.946+0000 7f507420f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.959+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50680017f0 con 0x7f506c0a98f0 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.959+0000 7f5070fa9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.959+0000 7f5070fa9640 1 -- 172.21.15.32:0/3285911248 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5068001c00 con 0x7f506c0a98f0 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.959+0000 7f5070fa9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.959+0000 7f5070fa9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.959+0000 7f5070fa9640 10 monclient: dump: 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:30.961 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:30.962 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:30.962 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:30.962 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:30.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.959+0000 7f507420f640 10 monclient: get_monmap_and_config success 2024-03-25T18:44:30.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.959+0000 7f507420f640 4 set_mon_vals no callback set 2024-03-25T18:44:30.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.960+0000 7f507420f640 10 monclient: shutdown 2024-03-25T18:44:30.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.960+0000 7f507420f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:30.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.960+0000 7f507420f640 1 -- 172.21.15.32:0/3285911248 >> v1:172.21.15.32:6789/0 conn(0x7f506c0a98f0 legacy=0x7f506c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:30.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.960+0000 7f507420f640 1 -- 172.21.15.32:0/3285911248 shutdown_connections 2024-03-25T18:44:30.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.960+0000 7f507420f640 1 -- 172.21.15.32:0/3285911248 >> 172.21.15.32:0/3285911248 conn(0x7f506c0b7ea0 msgr2=0x7f506c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:30.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.960+0000 7f507420f640 1 -- 172.21.15.32:0/3285911248 shutdown_connections 2024-03-25T18:44:30.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.960+0000 7f507420f640 1 -- 172.21.15.32:0/3285911248 wait complete. 2024-03-25T18:44:30.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.961+0000 7f507420f640 10 monclient: build_initial_monmap 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.961+0000 7f507420f640 10 monclient: monmap: 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 1 librados: starting msgr at 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 1 librados: starting objecter 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 1 Processor -- start 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 1 -- start start 2024-03-25T18:44:30.965 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 1 librados: setting wanted keys 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 1 librados: calling monclient init 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 10 monclient: init 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 10 monclient(hunting): picked mon.noname-b con 0x7f506c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.962+0000 7f507420f640 10 monclient(hunting): picked mon.noname-a con 0x7f506c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.963+0000 7f507420f640 10 monclient(hunting): picked mon.noname-c con 0x7f506c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.963+0000 7f507420f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c1369b0 con 0x7f506c13f710 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.963+0000 7f507420f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c1366b0 con 0x7f506c0a98f0 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.963+0000 7f507420f640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f506c133ad0 con 0x7f506c1430c0 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.963+0000 7f507420f640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:30.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.963+0000 7f507420f640 10 monclient(hunting): authenticate will time out at 6377.152600s 2024-03-25T18:44:30.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.975+0000 7f50727ac640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f506c1430c0 0x7f506c1333c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:47942/0 (socket says 172.21.15.32:47942) 2024-03-25T18:44:30.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.975+0000 7f50727ac640 1 -- 172.21.15.32:0/2586603591 learned_addr learned my addr 172.21.15.32:0/2586603591 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:30.978 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.977+0000 7f50627fc640 10 client.?.objecter ms_handle_connect 0x7f506c1430c0 2024-03-25T18:44:30.978 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.977+0000 7f50627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:30.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.981+0000 7f50627fc640 10 client.?.objecter ms_handle_connect 0x7f506c0a98f0 2024-03-25T18:44:30.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.981+0000 7f50627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:30.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.988+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590663059 0 0) 0x7f506c1366b0 con 0x7f506c0a98f0 2024-03-25T18:44:30.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.988+0000 7f50627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:44:30.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.988+0000 7f50627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:30.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.988+0000 7f50627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:30.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.988+0000 7f50627fc640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:44:30.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.988+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5050004eb0 con 0x7f506c0a98f0 2024-03-25T18:44:30.992 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.991+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2683068765 0 0) 0x7f506c133ad0 con 0x7f506c1430c0 2024-03-25T18:44:30.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.991+0000 7f50627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:44:30.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.991+0000 7f50627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:30.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.991+0000 7f50627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:30.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.991+0000 7f50627fc640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:44:30.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.991+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506c1366b0 con 0x7f506c1430c0 2024-03-25T18:44:30.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.992+0000 7f50627fc640 10 client.?.objecter ms_handle_connect 0x7f506c13f710 2024-03-25T18:44:30.993 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.992+0000 7f50627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:30.999 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.998+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456964866 0 0) 0x7f506c1369b0 con 0x7f506c13f710 2024-03-25T18:44:30.999 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.998+0000 7f50627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:44:30.999 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.998+0000 7f50627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:31.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.998+0000 7f50627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:31.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.998+0000 7f50627fc640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:44:31.000 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:30.998+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f506c133ad0 con 0x7f506c13f710 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.000+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4281223533 0 0) 0x7f506c1366b0 con 0x7f506c1430c0 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.000+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f506c1369b0 con 0x7f506c1430c0 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.000+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5064001c10 con 0x7f506c1430c0 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.000+0000 7f50627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.000+0000 7f50627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.000+0000 7f50627fc640 10 monclient(hunting): dump: 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:31.002 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:31.003 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:31.003 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:31.003 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:31.003 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:31.003 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.000+0000 7f50627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:44:31.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.003+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811477710 0 0) 0x7f5050004eb0 con 0x7f506c0a98f0 2024-03-25T18:44:31.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.003+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f506c1366b0 con 0x7f506c0a98f0 2024-03-25T18:44:31.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.003+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5068001b00 con 0x7f506c0a98f0 2024-03-25T18:44:31.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.003+0000 7f50627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:31.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.003+0000 7f50627fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:31.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.003+0000 7f50627fc640 10 monclient(hunting): dump: 2024-03-25T18:44:31.004 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:31.005 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:31.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.011+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1531845171 0 0) 0x7f506c133ad0 con 0x7f506c13f710 2024-03-25T18:44:31.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.011+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5050004eb0 con 0x7f506c13f710 2024-03-25T18:44:31.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.011+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f505c0017f0 con 0x7f506c13f710 2024-03-25T18:44:31.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.011+0000 7f50627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:31.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.011+0000 7f50627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:44:31.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.011+0000 7f50627fc640 10 monclient(hunting): dump: 2024-03-25T18:44:31.012 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:31.013 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3983371764 0 0) 0x7f506c1366b0 con 0x7f506c0a98f0 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 >> v1:172.21.15.32:6790/0 conn(0x7f506c1430c0 legacy=0x7f506c1333c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 >> v1:172.21.15.32:6789/0 conn(0x7f506c13f710 legacy=0x7f506c132cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 1 monclient: found mon.b 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f506c136e40 con 0x7f506c0a98f0 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 10 monclient: _finish_auth 0 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f50627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:31.020 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 5 monclient: authenticate success, global_id 4450 2024-03-25T18:44:31.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 10 monclient: _renew_subs 2024-03-25T18:44:31.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:31.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f506c1378f0 con 0x7f506c0a98f0 2024-03-25T18:44:31.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:44:31.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 10 monclient: _renew_subs 2024-03-25T18:44:31.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:31.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f506c0b1480 con 0x7f506c0a98f0 2024-03-25T18:44:31.021 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.018+0000 7f507420f640 1 librados: init done 2024-03-25T18:44:31.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.020+0000 7f507420f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:31.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.020+0000 7f507420f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:31.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.020+0000 7f507420f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:31.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.020+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f506c137030 con 0x7f506c0a98f0 2024-03-25T18:44:31.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.304+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50680038a0 con 0x7f506c0a98f0 2024-03-25T18:44:31.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.304+0000 7f50627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:31.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.304+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5068003bb0 con 0x7f506c0a98f0 2024-03-25T18:44:31.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.304+0000 7f50627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:31.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.304+0000 7f50627fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:31.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.304+0000 7f50627fc640 10 monclient: dump: 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:31.306 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.304+0000 7f5063fff640 4 set_mon_vals no callback set 2024-03-25T18:44:31.307 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.304+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5068019b80 con 0x7f506c0a98f0 2024-03-25T18:44:31.307 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.305+0000 7f50717aa640 10 monclient: get_auth_request con 0x7f505006e160 auth_method 0 2024-03-25T18:44:31.307 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.305+0000 7f50627fc640 10 client.4450.objecter ms_handle_connect 0x7f505006e160 2024-03-25T18:44:31.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.313+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5068053620 con 0x7f506c0a98f0 2024-03-25T18:44:31.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.313+0000 7f50627fc640 10 client.4450.objecter ms_dispatch 0x7f506c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:44:31.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.313+0000 7f50627fc640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:44:31.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.313+0000 7f50627fc640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:44:31.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.313+0000 7f50627fc640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-25T18:44:31.316 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.316+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f506804f130 con 0x7f506c0a98f0 2024-03-25T18:44:31.317 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.316+0000 7f50627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:31.317 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.316+0000 7f50627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:44:31.552 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.551+0000 7f507420f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:31.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.551+0000 7f507420f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:31.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.551+0000 7f507420f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:31.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.551+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f506c137b30 con 0x7f506c0a98f0 2024-03-25T18:44:31.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.571+0000 7f50627fc640 1 -- 172.21.15.32:0/2586603591 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5068051060 con 0x7f506c0a98f0 2024-03-25T18:44:31.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.571+0000 7f50627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:31.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.572+0000 7f50627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:44:31.573 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:44:31.573 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 10 librados: watch_flush enter 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 10 librados: watch_flush exit 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 10 client.4450.objecter successfully canceled tick 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 >> v1:172.21.15.136:6800/1141502808 conn(0x7f505006e160 legacy=0x7f5050070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 10 monclient: shutdown 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 >> v1:172.21.15.136:6789/0 conn(0x7f506c0a98f0 legacy=0x7f506c1371e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 shutdown_connections 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 >> 172.21.15.32:0/2586603591 conn(0x7f506c0b7ea0 msgr2=0x7f506c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:31.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 shutdown_connections 2024-03-25T18:44:31.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 1 -- 172.21.15.32:0/2586603591 wait complete. 2024-03-25T18:44:31.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:31.574+0000 7f507420f640 1 librados: shutdown 2024-03-25T18:44:31.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:44:38.394 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:44:38.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.495+0000 7f532c60d640 10 monclient: get_monmap_and_config 2024-03-25T18:44:38.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.495+0000 7f532c60d640 10 monclient: build_initial_monmap 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.495+0000 7f532c60d640 10 monclient: monmap: 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:44:38.495708+0000 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:44:38.495708+0000 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:44:38.496 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:38.497 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.495+0000 7f532c60d640 1 Processor -- start 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.495+0000 7f532c60d640 1 -- start start 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.495+0000 7f532c60d640 10 monclient: init 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 10 monclient(hunting): picked mon.noname-b con 0x7f53240b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 10 monclient(hunting): picked mon.noname-a con 0x7f53240b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 10 monclient(hunting): picked mon.noname-c con 0x7f532414be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53240af940 con 0x7f53240b1730 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53240b0a70 con 0x7f532414be90 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53240b0d50 con 0x7f53240b4d60 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.496+0000 7f532c60d640 10 monclient(hunting): authenticate will time out at 6384.685219s 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.497+0000 7f532abaa640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f532414be90 0x7f532414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:51402/0 (socket says 172.21.15.32:51402) 2024-03-25T18:44:38.498 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.497+0000 7f532abaa640 1 -- 172.21.15.32:0/1186736125 learned_addr learned my addr 172.21.15.32:0/1186736125 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:38.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.516+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 842872685 0 0) 0x7f53240b0a70 con 0x7f532414be90 2024-03-25T18:44:38.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.516+0000 7f53293a7640 10 monclient(hunting): _negotiate 2024-03-25T18:44:38.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.516+0000 7f53293a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:38.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.516+0000 7f53293a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:38.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.516+0000 7f53293a7640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:44:38.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.516+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5310003520 con 0x7f532414be90 2024-03-25T18:44:38.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.523+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439223457 0 0) 0x7f53240af940 con 0x7f53240b1730 2024-03-25T18:44:38.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.523+0000 7f53293a7640 10 monclient(hunting): _negotiate 2024-03-25T18:44:38.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.523+0000 7f53293a7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:38.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.523+0000 7f53293a7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:38.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.523+0000 7f53293a7640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:44:38.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.523+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53240b0a70 con 0x7f53240b1730 2024-03-25T18:44:38.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2239034132 0 0) 0x7f53240b0a70 con 0x7f53240b1730 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 >> v1:172.21.15.32:6790/0 conn(0x7f53240b4d60 legacy=0x7f532413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 >> v1:172.21.15.136:6789/0 conn(0x7f532414be90 legacy=0x7f532414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 10 monclient: _finish_hunting 0 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 1 monclient: found mon.noname-a 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53240b1030 con 0x7f53240b1730 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 10 monclient: _finish_auth 0 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 10 monclient: _check_auth_tickets 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f53293a7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f532c60d640 5 monclient: authenticate success, global_id 4470 2024-03-25T18:44:38.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.538+0000 7f532c60d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:38.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.541+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5314001630 con 0x7f53240b1730 2024-03-25T18:44:38.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.541+0000 7f53293a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:38.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.541+0000 7f53293a7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.541+0000 7f53293a7640 10 monclient: dump: 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:38.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.541+0000 7f532c60d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:38.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.543+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53140018c0 con 0x7f53240b1730 2024-03-25T18:44:38.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.543+0000 7f53293a7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.543+0000 7f53293a7640 1 -- 172.21.15.32:0/1186736125 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5314001bd0 con 0x7f53240b1730 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.543+0000 7f53293a7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.543+0000 7f53293a7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.543+0000 7f53293a7640 10 monclient: dump: 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:38.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.543+0000 7f532c60d640 10 monclient: get_monmap_and_config success 2024-03-25T18:44:38.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.543+0000 7f532c60d640 4 set_mon_vals no callback set 2024-03-25T18:44:38.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.544+0000 7f532c60d640 10 monclient: shutdown 2024-03-25T18:44:38.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.544+0000 7f532c60d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:38.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.544+0000 7f532c60d640 1 -- 172.21.15.32:0/1186736125 >> v1:172.21.15.32:6789/0 conn(0x7f53240b1730 legacy=0x7f53240b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.544+0000 7f532c60d640 1 -- 172.21.15.32:0/1186736125 shutdown_connections 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.544+0000 7f532c60d640 1 -- 172.21.15.32:0/1186736125 >> 172.21.15.32:0/1186736125 conn(0x7f53240617a0 msgr2=0x7f5324061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.545+0000 7f532c60d640 1 -- 172.21.15.32:0/1186736125 shutdown_connections 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.545+0000 7f532c60d640 1 -- 172.21.15.32:0/1186736125 wait complete. 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.545+0000 7f532c60d640 10 monclient: build_initial_monmap 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.545+0000 7f532c60d640 10 monclient: monmap: 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:38.546 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:44:38.547 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:44:38.547 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:38.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.546+0000 7f532c60d640 1 librados: starting msgr at 2024-03-25T18:44:38.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.546+0000 7f532c60d640 1 librados: starting objecter 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.546+0000 7f532c60d640 1 Processor -- start 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.546+0000 7f532c60d640 1 -- start start 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.546+0000 7f532c60d640 1 librados: setting wanted keys 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.546+0000 7f532c60d640 1 librados: calling monclient init 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.546+0000 7f532c60d640 10 monclient: init 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 10 monclient(hunting): picked mon.noname-a con 0x7f53240b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 10 monclient(hunting): picked mon.noname-c con 0x7f53240b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 10 monclient(hunting): picked mon.noname-b con 0x7f532414be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5324138440 con 0x7f53240b1730 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53241325b0 con 0x7f532414be90 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5324132740 con 0x7f53240b4d60 2024-03-25T18:44:38.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:38.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.547+0000 7f532c60d640 10 monclient(hunting): authenticate will time out at 6384.735801s 2024-03-25T18:44:38.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.557+0000 7f5329ba8640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f53240b4d60 0x7f532413b340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:47958/0 (socket says 172.21.15.32:47958) 2024-03-25T18:44:38.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.557+0000 7f5329ba8640 1 -- 172.21.15.32:0/3248360471 learned_addr learned my addr 172.21.15.32:0/3248360471 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:38.559 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.560+0000 7f531affd640 10 client.?.objecter ms_handle_connect 0x7f53240b4d60 2024-03-25T18:44:38.560 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.560+0000 7f531affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:38.562 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.561+0000 7f531affd640 10 client.?.objecter ms_handle_connect 0x7f53240b1730 2024-03-25T18:44:38.562 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.561+0000 7f531affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:38.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.565+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 951527803 0 0) 0x7f5324132740 con 0x7f53240b4d60 2024-03-25T18:44:38.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.565+0000 7f531affd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:38.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.565+0000 7f531affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:38.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.565+0000 7f531affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:38.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.565+0000 7f531affd640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:44:38.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.565+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5308004eb0 con 0x7f53240b4d60 2024-03-25T18:44:38.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.572+0000 7f531affd640 10 client.?.objecter ms_handle_connect 0x7f532414be90 2024-03-25T18:44:38.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.572+0000 7f531affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:38.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.576+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1160482897 0 0) 0x7f5324138440 con 0x7f53240b1730 2024-03-25T18:44:38.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.576+0000 7f531affd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:38.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.576+0000 7f531affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:38.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.576+0000 7f531affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:38.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.577+0000 7f531affd640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:44:38.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.577+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5324132740 con 0x7f53240b1730 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.586+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1747098595 0 0) 0x7f5308004eb0 con 0x7f53240b4d60 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.586+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5324138440 con 0x7f53240b4d60 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.586+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f53140016f0 con 0x7f53240b4d60 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.586+0000 7f531affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.586+0000 7f531affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.586+0000 7f531affd640 10 monclient(hunting): dump: 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:38.587 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.586+0000 7f531affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.588+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 873789094 0 0) 0x7f53241325b0 con 0x7f532414be90 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.588+0000 7f531affd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.588+0000 7f531affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.588+0000 7f531affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.588+0000 7f531affd640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:44:38.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.588+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5308004eb0 con 0x7f532414be90 2024-03-25T18:44:38.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3338322385 0 0) 0x7f5324138440 con 0x7f53240b4d60 2024-03-25T18:44:38.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 >> v1:172.21.15.136:6789/0 conn(0x7f532414be90 legacy=0x7f53241e3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:38.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 >> v1:172.21.15.32:6789/0 conn(0x7f53240b1730 legacy=0x7f5324131ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 10 monclient: _finish_hunting 0 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 1 monclient: found mon.c 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53241328d0 con 0x7f53240b4d60 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 10 monclient: _finish_auth 0 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 10 monclient: _check_auth_tickets 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f531affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 5 monclient: authenticate success, global_id 4460 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 10 monclient: _renew_subs 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5324139800 con 0x7f53240b4d60 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 10 client.4460.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 10 monclient: _renew_subs 2024-03-25T18:44:38.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:44:38.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5324139040 con 0x7f53240b4d60 2024-03-25T18:44:38.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.599+0000 7f532c60d640 1 librados: init done 2024-03-25T18:44:38.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.600+0000 7f532c60d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:38.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.600+0000 7f532c60d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:38.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.600+0000 7f532c60d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:44:38.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.600+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f532413b140 con 0x7f53240b4d60 2024-03-25T18:44:38.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.923+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5314001220 con 0x7f53240b4d60 2024-03-25T18:44:38.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.923+0000 7f531affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:38.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.923+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5314003b00 con 0x7f53240b4d60 2024-03-25T18:44:38.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.923+0000 7f531affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:38.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.923+0000 7f531affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:44:38.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.923+0000 7f531affd640 10 monclient: dump: 2024-03-25T18:44:38.923 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:38.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.923+0000 7f5328ba6640 4 set_mon_vals no callback set 2024-03-25T18:44:38.925 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.924+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5314003d20 con 0x7f53240b4d60 2024-03-25T18:44:38.925 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.924+0000 7f532a3a9640 10 monclient: get_auth_request con 0x7f530806de00 auth_method 0 2024-03-25T18:44:38.925 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.925+0000 7f531affd640 10 client.4460.objecter ms_handle_connect 0x7f530806de00 2024-03-25T18:44:38.930 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.930+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5314054960 con 0x7f53240b4d60 2024-03-25T18:44:38.930 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.930+0000 7f531affd640 10 client.4460.objecter ms_dispatch 0x7f5324000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:44:38.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.930+0000 7f531affd640 3 client.4460.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:44:38.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.930+0000 7f531affd640 3 client.4460.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:44:38.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.930+0000 7f531affd640 20 client.4460.objecter dump_active .. 0 homeless 2024-03-25T18:44:38.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.930+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f53140502a0 con 0x7f53240b4d60 2024-03-25T18:44:38.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.930+0000 7f531affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:38.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:38.930+0000 7f531affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:44:39.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.184+0000 7f532c60d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:39.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.185+0000 7f532c60d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:39.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.185+0000 7f532c60d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:44:39.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.185+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53240627b0 con 0x7f53240b4d60 2024-03-25T18:44:39.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.207+0000 7f531affd640 1 -- 172.21.15.32:0/3248360471 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f53140523a0 con 0x7f53240b4d60 2024-03-25T18:44:39.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.207+0000 7f531affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:39.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.207+0000 7f531affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:44:39.207 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:44:39.207 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 10 librados: watch_flush enter 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 10 librados: watch_flush exit 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 20 client.4460.objecter shutdown clearing up homeless session... 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 10 client.4460.objecter successfully canceled tick 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 >> v1:172.21.15.136:6800/1141502808 conn(0x7f530806de00 legacy=0x7f53080702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 10 monclient: shutdown 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 >> v1:172.21.15.32:6790/0 conn(0x7f53240b4d60 legacy=0x7f532413b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 shutdown_connections 2024-03-25T18:44:39.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 >> 172.21.15.32:0/3248360471 conn(0x7f53240617a0 msgr2=0x7f53240b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:39.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.209+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 shutdown_connections 2024-03-25T18:44:39.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.210+0000 7f532c60d640 1 -- 172.21.15.32:0/3248360471 wait complete. 2024-03-25T18:44:39.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:39.210+0000 7f532c60d640 1 librados: shutdown 2024-03-25T18:44:39.222 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:44:46.028 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.148+0000 7f9e046fb640 10 monclient: get_monmap_and_config 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.148+0000 7f9e046fb640 10 monclient: build_initial_monmap 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.148+0000 7f9e046fb640 10 monclient: monmap: 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:44:46.149423+0000 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:44:46.149423+0000 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:46.149 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:46.150 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:44:46.150 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:44:46.150 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:46.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.149+0000 7f9e046fb640 1 Processor -- start 2024-03-25T18:44:46.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.150+0000 7f9e046fb640 1 -- start start 2024-03-25T18:44:46.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.150+0000 7f9e046fb640 10 monclient: init 2024-03-25T18:44:46.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:46.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:44:46.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 10 monclient(hunting): picked mon.noname-c con 0x7f9dfc0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:46.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 10 monclient(hunting): picked mon.noname-a con 0x7f9dfc13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:46.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 10 monclient(hunting): picked mon.noname-b con 0x7f9dfc1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:46.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc142520 con 0x7f9dfc13f710 2024-03-25T18:44:46.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc0b28a0 con 0x7f9dfc0a98f0 2024-03-25T18:44:46.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc0b2a30 con 0x7f9dfc1430c0 2024-03-25T18:44:46.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:46.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.151+0000 7f9e046fb640 10 monclient(hunting): authenticate will time out at 6392.339787s 2024-03-25T18:44:46.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.159+0000 7f9e02497640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9dfc0a98f0 0x7f9dfc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:56076/0 (socket says 172.21.15.32:56076) 2024-03-25T18:44:46.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.159+0000 7f9e02497640 1 -- 172.21.15.32:0/3860030490 learned_addr learned my addr 172.21.15.32:0/3860030490 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:46.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.176+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325607070 0 0) 0x7f9dfc0b28a0 con 0x7f9dfc0a98f0 2024-03-25T18:44:46.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.176+0000 7f9e01495640 10 monclient(hunting): _negotiate 2024-03-25T18:44:46.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.176+0000 7f9e01495640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:46.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.176+0000 7f9e01495640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:46.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.176+0000 7f9e01495640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:44:46.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.176+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de8003520 con 0x7f9dfc0a98f0 2024-03-25T18:44:46.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.181+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1703778992 0 0) 0x7f9dfc0b2a30 con 0x7f9dfc1430c0 2024-03-25T18:44:46.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.181+0000 7f9e01495640 10 monclient(hunting): _negotiate 2024-03-25T18:44:46.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.181+0000 7f9e01495640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:46.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.181+0000 7f9e01495640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:46.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.181+0000 7f9e01495640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:44:46.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.181+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc0b28a0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.187+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2640155701 0 0) 0x7f9dfc142520 con 0x7f9dfc13f710 2024-03-25T18:44:46.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.187+0000 7f9e01495640 10 monclient(hunting): _negotiate 2024-03-25T18:44:46.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.187+0000 7f9e01495640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:46.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.187+0000 7f9e01495640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:46.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.187+0000 7f9e01495640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:44:46.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.187+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dfc0b2a30 con 0x7f9dfc13f710 2024-03-25T18:44:46.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3999231468 0 0) 0x7f9dfc0b2a30 con 0x7f9dfc13f710 2024-03-25T18:44:46.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 >> v1:172.21.15.32:6790/0 conn(0x7f9dfc1430c0 legacy=0x7f9dfc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:46.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 >> v1:172.21.15.136:6789/0 conn(0x7f9dfc0a98f0 legacy=0x7f9dfc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:46.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 10 monclient: _finish_hunting 0 2024-03-25T18:44:46.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 1 monclient: found mon.noname-a 2024-03-25T18:44:46.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:46.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:44:46.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc0b44c0 con 0x7f9dfc13f710 2024-03-25T18:44:46.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 10 monclient: _finish_auth 0 2024-03-25T18:44:46.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 10 monclient: _check_auth_tickets 2024-03-25T18:44:46.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:46.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9dec0015b0 con 0x7f9dfc13f710 2024-03-25T18:44:46.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:46.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:44:46.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e01495640 10 monclient: dump: 2024-03-25T18:44:46.202 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:46.203 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:46.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e046fb640 5 monclient: authenticate success, global_id 4476 2024-03-25T18:44:46.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.200+0000 7f9e046fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.209+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dec001710 con 0x7f9dfc13f710 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.209+0000 7f9e01495640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.209+0000 7f9e01495640 1 -- 172.21.15.32:0/3860030490 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9dec001b60 con 0x7f9dfc13f710 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.209+0000 7f9e01495640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.209+0000 7f9e01495640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.209+0000 7f9e01495640 10 monclient: dump: 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:46.209 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.209+0000 7f9e046fb640 10 monclient: get_monmap_and_config success 2024-03-25T18:44:46.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.209+0000 7f9e046fb640 4 set_mon_vals no callback set 2024-03-25T18:44:46.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.210+0000 7f9e046fb640 10 monclient: shutdown 2024-03-25T18:44:46.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.210+0000 7f9e046fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:46.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.210+0000 7f9e046fb640 1 -- 172.21.15.32:0/3860030490 >> v1:172.21.15.32:6789/0 conn(0x7f9dfc13f710 legacy=0x7f9dfc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:46.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.210+0000 7f9e046fb640 1 -- 172.21.15.32:0/3860030490 shutdown_connections 2024-03-25T18:44:46.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.210+0000 7f9e046fb640 1 -- 172.21.15.32:0/3860030490 >> 172.21.15.32:0/3860030490 conn(0x7f9dfc0b7ea0 msgr2=0x7f9dfc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:46.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.210+0000 7f9e046fb640 1 -- 172.21.15.32:0/3860030490 shutdown_connections 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.210+0000 7f9e046fb640 1 -- 172.21.15.32:0/3860030490 wait complete. 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 10 monclient: build_initial_monmap 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 10 monclient: monmap: 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 1 librados: starting msgr at 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 1 librados: starting objecter 2024-03-25T18:44:46.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 1 Processor -- start 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 1 -- start start 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 1 librados: setting wanted keys 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 1 librados: calling monclient init 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.211+0000 7f9e046fb640 10 monclient: init 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 10 monclient(hunting): picked mon.noname-c con 0x7f9dfc0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 10 monclient(hunting): picked mon.noname-a con 0x7f9dfc13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 10 monclient(hunting): picked mon.noname-b con 0x7f9dfc1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc0abb00 con 0x7f9dfc13f710 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc0abde0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dfc14eab0 con 0x7f9dfc0a98f0 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:46.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.212+0000 7f9e046fb640 10 monclient(hunting): authenticate will time out at 6392.400879s 2024-03-25T18:44:46.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.220+0000 7f9e02c98640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9dfc1430c0 0x7f9dfc1d76b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:56124/0 (socket says 172.21.15.32:56124) 2024-03-25T18:44:46.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.220+0000 7f9e02c98640 1 -- 172.21.15.32:0/3891854601 learned_addr learned my addr 172.21.15.32:0/3891854601 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:46.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.221+0000 7f9df2ffd640 10 client.?.objecter ms_handle_connect 0x7f9dfc1430c0 2024-03-25T18:44:46.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.221+0000 7f9df2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:46.235 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.235+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2528227882 0 0) 0x7f9dfc0abde0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.235 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.235+0000 7f9df2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:46.235 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.235+0000 7f9df2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:46.236 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.235+0000 7f9df2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:46.236 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.235+0000 7f9df2ffd640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:44:46.236 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.235+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9de0004eb0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.243+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 582365246 0 0) 0x7f9de0004eb0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.243+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dfc0abde0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.243+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9df4001c90 con 0x7f9dfc1430c0 2024-03-25T18:44:46.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.243+0000 7f9df2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:46.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.243+0000 7f9df2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:44:46.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.243+0000 7f9df2ffd640 10 monclient(hunting): dump: 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:46.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.243+0000 7f9df2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:44:46.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.247+0000 7f9df2ffd640 10 client.?.objecter ms_handle_connect 0x7f9dfc13f710 2024-03-25T18:44:46.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.247+0000 7f9df2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:46.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.251+0000 7f9df2ffd640 10 client.?.objecter ms_handle_connect 0x7f9dfc0a98f0 2024-03-25T18:44:46.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.251+0000 7f9df2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1485970973 0 0) 0x7f9dfc0abde0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 >> v1:172.21.15.32:6790/0 conn(0x7f9dfc0a98f0 legacy=0x7f9dfc138890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 >> v1:172.21.15.32:6789/0 conn(0x7f9dfc13f710 legacy=0x7f9dfc133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 1 monclient: found mon.b 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc14e7a0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:44:46.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9df2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 10 monclient: _renew_subs 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dfc138fa0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 10 monclient: _renew_subs 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dfc134890 con 0x7f9dfc1430c0 2024-03-25T18:44:46.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.254+0000 7f9e046fb640 1 librados: init done 2024-03-25T18:44:46.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.255+0000 7f9e046fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:46.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.255+0000 7f9e046fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:46.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.255+0000 7f9e046fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:46.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.255+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dfc14ec40 con 0x7f9dfc1430c0 2024-03-25T18:44:46.535 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.534+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9df4003970 con 0x7f9dfc1430c0 2024-03-25T18:44:46.535 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.534+0000 7f9df2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:46.535 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.534+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9df4003d80 con 0x7f9dfc1430c0 2024-03-25T18:44:46.536 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.534+0000 7f9df2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:46.536 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.534+0000 7f9df2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:46.536 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.534+0000 7f9df2ffd640 10 monclient: dump: 2024-03-25T18:44:46.536 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:46.536 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:46.536 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:46.536 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:46.537 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:46.537 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:46.537 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:46.537 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:46.537 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:46.537 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:46.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.534+0000 7f9e00c94640 4 set_mon_vals no callback set 2024-03-25T18:44:46.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.535+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f9df4003fa0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.535+0000 7f9e02497640 10 monclient: get_auth_request con 0x7f9de006dc20 auth_method 0 2024-03-25T18:44:46.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.536+0000 7f9df2ffd640 10 client.4462.objecter ms_handle_connect 0x7f9de006dc20 2024-03-25T18:44:46.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.539+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f9df4054b00 con 0x7f9dfc1430c0 2024-03-25T18:44:46.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.539+0000 7f9df2ffd640 10 client.4462.objecter ms_dispatch 0x7f9dfc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:44:46.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.539+0000 7f9df2ffd640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:44:46.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.539+0000 7f9df2ffd640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:44:46.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.539+0000 7f9df2ffd640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T18:44:46.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.539+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9df4050610 con 0x7f9dfc1430c0 2024-03-25T18:44:46.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.539+0000 7f9df2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:46.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.539+0000 7f9df2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:44:46.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.794+0000 7f9e046fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:46.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.794+0000 7f9e046fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:46.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.794+0000 7f9e046fb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:46.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.794+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dfc0627b0 con 0x7f9dfc1430c0 2024-03-25T18:44:46.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.840+0000 7f9df2ffd640 1 -- 172.21.15.32:0/3891854601 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f9df4052540 con 0x7f9dfc1430c0 2024-03-25T18:44:46.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.840+0000 7f9df2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:46.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.840+0000 7f9df2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:44:46.841 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:44:46.841 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:44:46.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.842+0000 7f9e046fb640 10 librados: watch_flush enter 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.842+0000 7f9e046fb640 10 librados: watch_flush exit 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.842+0000 7f9e046fb640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.842+0000 7f9e046fb640 10 client.4462.objecter successfully canceled tick 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 >> v1:172.21.15.136:6800/1141502808 conn(0x7f9de006dc20 legacy=0x7f9de00700e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 10 monclient: shutdown 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 >> v1:172.21.15.136:6789/0 conn(0x7f9dfc1430c0 legacy=0x7f9dfc1d76b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 shutdown_connections 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 >> 172.21.15.32:0/3891854601 conn(0x7f9dfc0b7ea0 msgr2=0x7f9dfc13d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 shutdown_connections 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 1 -- 172.21.15.32:0/3891854601 wait complete. 2024-03-25T18:44:46.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:46.843+0000 7f9e046fb640 1 librados: shutdown 2024-03-25T18:44:46.856 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:44:47.969 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:00.297 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:47.977 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:00.304 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:48.010 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:00.338 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:48.019 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:00.347 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:48.035 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:00.363 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:48.035 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:00.352 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:48.047 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:00.364 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:48.068 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:00.396 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:48.073 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:00.390 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:44:53.664 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:44:53.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.782+0000 7fb7423af640 10 monclient: get_monmap_and_config 2024-03-25T18:44:53.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.782+0000 7fb7423af640 10 monclient: build_initial_monmap 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.782+0000 7fb7423af640 10 monclient: monmap: 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:44:53.782966+0000 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:44:53.782966+0000 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:44:53.783 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:53.784 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.783+0000 7fb7423af640 1 Processor -- start 2024-03-25T18:44:53.784 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.783+0000 7fb7423af640 1 -- start start 2024-03-25T18:44:53.784 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.783+0000 7fb7423af640 10 monclient: init 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.783+0000 7fb7423af640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.783+0000 7fb7423af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.783+0000 7fb7423af640 10 monclient(hunting): picked mon.noname-c con 0x7fb73c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.783+0000 7fb7423af640 10 monclient(hunting): picked mon.noname-a con 0x7fb73c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.783+0000 7fb7423af640 10 monclient(hunting): picked mon.noname-b con 0x7fb73c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.784+0000 7fb7423af640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c142520 con 0x7fb73c13f710 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.784+0000 7fb7423af640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c0b28a0 con 0x7fb73c0a98f0 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.784+0000 7fb7423af640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c0b2a30 con 0x7fb73c1430c0 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.784+0000 7fb7423af640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:53.785 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.784+0000 7fb7423af640 10 monclient(hunting): authenticate will time out at 6399.972556s 2024-03-25T18:44:53.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.798+0000 7fb73b7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fb73c13f710 0x7fb73c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:35574/0 (socket says 172.21.15.32:35574) 2024-03-25T18:44:53.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.798+0000 7fb73b7fe640 1 -- 172.21.15.32:0/3113908243 learned_addr learned my addr 172.21.15.32:0/3113908243 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:53.816 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.816+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888404831 0 0) 0x7fb73c0b2a30 con 0x7fb73c1430c0 2024-03-25T18:44:53.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.816+0000 7fb73affd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:53.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.816+0000 7fb73affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:53.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.816+0000 7fb73affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:53.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.816+0000 7fb73affd640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:44:53.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.816+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb724003520 con 0x7fb73c1430c0 2024-03-25T18:44:53.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.817+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974413922 0 0) 0x7fb73c0b28a0 con 0x7fb73c0a98f0 2024-03-25T18:44:53.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.817+0000 7fb73affd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:53.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.817+0000 7fb73affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:53.818 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.817+0000 7fb73affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:53.818 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.817+0000 7fb73affd640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:44:53.818 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.817+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73c0b2a30 con 0x7fb73c0a98f0 2024-03-25T18:44:53.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.823+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582777959 0 0) 0x7fb73c142520 con 0x7fb73c13f710 2024-03-25T18:44:53.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.823+0000 7fb73affd640 10 monclient(hunting): _negotiate 2024-03-25T18:44:53.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.823+0000 7fb73affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:53.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.823+0000 7fb73affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:53.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.823+0000 7fb73affd640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:44:53.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.823+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73c0b28a0 con 0x7fb73c13f710 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1025877088 0 0) 0x7fb724003520 con 0x7fb73c1430c0 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 >> v1:172.21.15.136:6789/0 conn(0x7fb73c0a98f0 legacy=0x7fb73c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 >> v1:172.21.15.32:6789/0 conn(0x7fb73c13f710 legacy=0x7fb73c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 10 monclient: _finish_hunting 0 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 1 monclient: found mon.noname-b 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c0b44c0 con 0x7fb73c1430c0 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 10 monclient: _finish_auth 0 2024-03-25T18:44:53.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 10 monclient: _check_auth_tickets 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb734001a10 con 0x7fb73c1430c0 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb73affd640 10 monclient: dump: 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:53.829 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:53.830 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:53.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb7423af640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:44:53.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.827+0000 7fb7423af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:44:53.847 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb734001b70 con 0x7fb73c1430c0 2024-03-25T18:44:53.847 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb73affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:53.847 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb73affd640 1 -- 172.21.15.32:0/3113908243 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb734001f80 con 0x7fb73c1430c0 2024-03-25T18:44:53.847 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb73affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:53.847 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb73affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb73affd640 10 monclient: dump: 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb7423af640 10 monclient: get_monmap_and_config success 2024-03-25T18:44:53.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb7423af640 4 set_mon_vals no callback set 2024-03-25T18:44:53.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb7423af640 10 monclient: shutdown 2024-03-25T18:44:53.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb7423af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:53.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.847+0000 7fb7423af640 1 -- 172.21.15.32:0/3113908243 >> v1:172.21.15.32:6790/0 conn(0x7fb73c1430c0 legacy=0x7fb73c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:53.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.848+0000 7fb7423af640 1 -- 172.21.15.32:0/3113908243 shutdown_connections 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.848+0000 7fb7423af640 1 -- 172.21.15.32:0/3113908243 >> 172.21.15.32:0/3113908243 conn(0x7fb73c0b7ea0 msgr2=0x7fb73c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.848+0000 7fb7423af640 1 -- 172.21.15.32:0/3113908243 shutdown_connections 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.848+0000 7fb7423af640 1 -- 172.21.15.32:0/3113908243 wait complete. 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.848+0000 7fb7423af640 10 monclient: build_initial_monmap 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.848+0000 7fb7423af640 10 monclient: monmap: 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:44:53.851 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:44:53.852 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:53.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.849+0000 7fb7423af640 1 librados: starting msgr at 2024-03-25T18:44:53.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.849+0000 7fb7423af640 1 librados: starting objecter 2024-03-25T18:44:53.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.849+0000 7fb7423af640 1 Processor -- start 2024-03-25T18:44:53.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.849+0000 7fb7423af640 1 -- start start 2024-03-25T18:44:53.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.849+0000 7fb7423af640 1 librados: setting wanted keys 2024-03-25T18:44:53.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.849+0000 7fb7423af640 1 librados: calling monclient init 2024-03-25T18:44:53.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.849+0000 7fb7423af640 10 monclient: init 2024-03-25T18:44:53.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 10 monclient: _reopen_session rank -1 2024-03-25T18:44:53.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:44:53.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 10 monclient(hunting): picked mon.noname-c con 0x7fb73c0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:44:53.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 10 monclient(hunting): picked mon.noname-b con 0x7fb73c13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:44:53.853 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 10 monclient(hunting): picked mon.noname-a con 0x7fb73c1430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:44:53.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c13b130 con 0x7fb73c1430c0 2024-03-25T18:44:53.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c139a20 con 0x7fb73c13f710 2024-03-25T18:44:53.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb73c139720 con 0x7fb73c0a98f0 2024-03-25T18:44:53.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 10 monclient(hunting): _renew_subs 2024-03-25T18:44:53.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.850+0000 7fb7423af640 10 monclient(hunting): authenticate will time out at 6400.038852s 2024-03-25T18:44:53.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.854+0000 7fb73b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb73c13f710 0x7fb73c14bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:35310/0 (socket says 172.21.15.32:35310) 2024-03-25T18:44:53.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.854+0000 7fb73b7fe640 1 -- 172.21.15.32:0/1988841042 learned_addr learned my addr 172.21.15.32:0/1988841042 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:44:53.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.859+0000 7fb738ff9640 10 client.?.objecter ms_handle_connect 0x7fb73c0a98f0 2024-03-25T18:44:53.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.859+0000 7fb738ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:53.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.859+0000 7fb738ff9640 10 client.?.objecter ms_handle_connect 0x7fb73c13f710 2024-03-25T18:44:53.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.859+0000 7fb738ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:53.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.863+0000 7fb738ff9640 10 client.?.objecter ms_handle_connect 0x7fb73c1430c0 2024-03-25T18:44:53.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.863+0000 7fb738ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:44:53.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.866+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2429115529 0 0) 0x7fb73c139a20 con 0x7fb73c13f710 2024-03-25T18:44:53.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.866+0000 7fb738ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:53.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.866+0000 7fb738ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:53.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.866+0000 7fb738ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:53.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.866+0000 7fb738ff9640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:44:53.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.866+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb728004eb0 con 0x7fb73c13f710 2024-03-25T18:44:53.873 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.872+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4009312879 0 0) 0x7fb73c139720 con 0x7fb73c0a98f0 2024-03-25T18:44:53.873 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.872+0000 7fb738ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:53.873 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.872+0000 7fb738ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:53.873 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.872+0000 7fb738ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:53.873 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.872+0000 7fb738ff9640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:44:53.873 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.872+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb73c139a20 con 0x7fb73c0a98f0 2024-03-25T18:44:53.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.875+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2155238611 0 0) 0x7fb728004eb0 con 0x7fb73c13f710 2024-03-25T18:44:53.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.875+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb73c139720 con 0x7fb73c13f710 2024-03-25T18:44:53.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.875+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb730001910 con 0x7fb73c13f710 2024-03-25T18:44:53.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.875+0000 7fb738ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:53.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.875+0000 7fb738ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:44:53.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.875+0000 7fb738ff9640 10 monclient(hunting): dump: 2024-03-25T18:44:53.875 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:53.875 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:53.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.875+0000 7fb738ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:44:53.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.882+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3868615824 0 0) 0x7fb73c0b44c0 con 0x7fb73c1430c0 2024-03-25T18:44:53.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.882+0000 7fb738ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:44:53.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.882+0000 7fb738ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:44:53.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.882+0000 7fb738ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:44:53.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.882+0000 7fb738ff9640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:44:53.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.882+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb728004eb0 con 0x7fb73c1430c0 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 285024567 0 0) 0x7fb73c139720 con 0x7fb73c13f710 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 >> v1:172.21.15.32:6790/0 conn(0x7fb73c0a98f0 legacy=0x7fb73c13a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 >> v1:172.21.15.32:6789/0 conn(0x7fb73c1430c0 legacy=0x7fb73c1e66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 1 monclient: found mon.b 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73c13b890 con 0x7fb73c13f710 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 10 monclient: _finish_auth 0 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:44:53.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb738ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 5 monclient: authenticate success, global_id 4468 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 10 monclient: _renew_subs 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb73c13bb70 con 0x7fb73c13f710 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 10 client.4468.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 10 monclient: _renew_subs 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb73c13c930 con 0x7fb73c13f710 2024-03-25T18:44:53.887 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.885+0000 7fb7423af640 1 librados: init done 2024-03-25T18:44:53.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.886+0000 7fb7423af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:53.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.886+0000 7fb7423af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:53.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.886+0000 7fb7423af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:53.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:53.886+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb73c13c5d0 con 0x7fb73c13f710 2024-03-25T18:44:54.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.182+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb7300047e0 con 0x7fb73c13f710 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.182+0000 7fb738ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.182+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb730004bf0 con 0x7fb73c13f710 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.182+0000 7fb738ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.182+0000 7fb738ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.182+0000 7fb738ff9640 10 monclient: dump: 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:44:54.183 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:44:54.184 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:44:54.184 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:44:54.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.182+0000 7fb73a7fc640 4 set_mon_vals no callback set 2024-03-25T18:44:54.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.183+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fb730004e10 con 0x7fb73c13f710 2024-03-25T18:44:54.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.184+0000 7fb73bfff640 10 monclient: get_auth_request con 0x7fb72806dd60 auth_method 0 2024-03-25T18:44:54.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.184+0000 7fb738ff9640 10 client.4468.objecter ms_handle_connect 0x7fb72806dd60 2024-03-25T18:44:54.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.188+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fb730054860 con 0x7fb73c13f710 2024-03-25T18:44:54.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.188+0000 7fb738ff9640 10 client.4468.objecter ms_dispatch 0x7fb73c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:44:54.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.188+0000 7fb738ff9640 3 client.4468.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:44:54.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.188+0000 7fb738ff9640 3 client.4468.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:44:54.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.188+0000 7fb738ff9640 20 client.4468.objecter dump_active .. 0 homeless 2024-03-25T18:44:54.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.199+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb730050370 con 0x7fb73c13f710 2024-03-25T18:44:54.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.199+0000 7fb738ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:44:54.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.199+0000 7fb738ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:44:54.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.414+0000 7fb7423af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:54.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.414+0000 7fb7423af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:54.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.414+0000 7fb7423af640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:44:54.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.414+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb73c13cbd0 con 0x7fb73c13f710 2024-03-25T18:44:54.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.456+0000 7fb738ff9640 1 -- 172.21.15.32:0/1988841042 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fb7300522a0 con 0x7fb73c13f710 2024-03-25T18:44:54.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.456+0000 7fb738ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:44:54.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.456+0000 7fb738ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:44:54.457 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:44:54.457 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:44:54.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 10 librados: watch_flush enter 2024-03-25T18:44:54.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 10 librados: watch_flush exit 2024-03-25T18:44:54.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 20 client.4468.objecter shutdown clearing up homeless session... 2024-03-25T18:44:54.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 10 client.4468.objecter successfully canceled tick 2024-03-25T18:44:54.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 >> v1:172.21.15.136:6800/1141502808 conn(0x7fb72806dd60 legacy=0x7fb728070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:54.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 10 monclient: shutdown 2024-03-25T18:44:54.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:44:54.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 >> v1:172.21.15.136:6789/0 conn(0x7fb73c13f710 legacy=0x7fb73c14bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:44:54.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 shutdown_connections 2024-03-25T18:44:54.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.458+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 >> 172.21.15.32:0/1988841042 conn(0x7fb73c0b7ea0 msgr2=0x7fb73c14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:44:54.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.459+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 shutdown_connections 2024-03-25T18:44:54.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.459+0000 7fb7423af640 1 -- 172.21.15.32:0/1988841042 wait complete. 2024-03-25T18:44:54.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:44:54.459+0000 7fb7423af640 1 librados: shutdown 2024-03-25T18:44:54.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:45:01.279 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:45:01.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.395+0000 7fb19eae6640 10 monclient: get_monmap_and_config 2024-03-25T18:45:01.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.395+0000 7fb19eae6640 10 monclient: build_initial_monmap 2024-03-25T18:45:01.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.395+0000 7fb19eae6640 10 monclient: monmap: 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:45:01.396788+0000 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:45:01.396788+0000 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:45:01.397 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:01.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.396+0000 7fb19eae6640 1 Processor -- start 2024-03-25T18:45:01.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.396+0000 7fb19eae6640 1 -- start start 2024-03-25T18:45:01.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.396+0000 7fb19eae6640 10 monclient: init 2024-03-25T18:45:01.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:01.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:45:01.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 10 monclient(hunting): picked mon.noname-b con 0x7fb1980b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:01.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1980b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:01.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 10 monclient(hunting): picked mon.noname-a con 0x7fb19814be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:01.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980af940 con 0x7fb19814be90 2024-03-25T18:45:01.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980b0a70 con 0x7fb1980b1730 2024-03-25T18:45:01.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1980b0d50 con 0x7fb1980b4d60 2024-03-25T18:45:01.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:01.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.397+0000 7fb19eae6640 10 monclient(hunting): authenticate will time out at 6407.587051s 2024-03-25T18:45:01.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.404+0000 7fb19c882640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fb1980b4d60 0x7fb19813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:36562/0 (socket says 172.21.15.32:36562) 2024-03-25T18:45:01.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.404+0000 7fb19c882640 1 -- 172.21.15.32:0/2417834589 learned_addr learned my addr 172.21.15.32:0/2417834589 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:01.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.412+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1260536311 0 0) 0x7fb1980b0d50 con 0x7fb1980b4d60 2024-03-25T18:45:01.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.412+0000 7fb1977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:01.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.412+0000 7fb1977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:01.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.412+0000 7fb1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:01.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.412+0000 7fb1977fe640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:45:01.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.412+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb188003520 con 0x7fb1980b4d60 2024-03-25T18:45:01.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1005137955 0 0) 0x7fb1980af940 con 0x7fb19814be90 2024-03-25T18:45:01.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1980b0d50 con 0x7fb19814be90 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872516129 0 0) 0x7fb1980b0a70 con 0x7fb1980b1730 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:45:01.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.427+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1980af940 con 0x7fb1980b1730 2024-03-25T18:45:01.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3126284437 0 0) 0x7fb188003520 con 0x7fb1980b4d60 2024-03-25T18:45:01.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 >> v1:172.21.15.136:6789/0 conn(0x7fb1980b1730 legacy=0x7fb1980b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 >> v1:172.21.15.32:6789/0 conn(0x7fb19814be90 legacy=0x7fb19814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 1 monclient: found mon.noname-b 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1980b1030 con 0x7fb1980b4d60 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 10 monclient: _finish_auth 0 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb18c001690 con 0x7fb1980b4d60 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb1977fe640 10 monclient: dump: 2024-03-25T18:45:01.431 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb19eae6640 5 monclient: authenticate success, global_id 4475 2024-03-25T18:45:01.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.429+0000 7fb19eae6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:01.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.450+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb18c0017f0 con 0x7fb1980b4d60 2024-03-25T18:45:01.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.450+0000 7fb1977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:01.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.450+0000 7fb1977fe640 1 -- 172.21.15.32:0/2417834589 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb18c001c00 con 0x7fb1980b4d60 2024-03-25T18:45:01.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.450+0000 7fb1977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:01.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.450+0000 7fb1977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:45:01.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.450+0000 7fb1977fe640 10 monclient: dump: 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.450+0000 7fb19eae6640 10 monclient: get_monmap_and_config success 2024-03-25T18:45:01.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.450+0000 7fb19eae6640 4 set_mon_vals no callback set 2024-03-25T18:45:01.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.451+0000 7fb19eae6640 10 monclient: shutdown 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.451+0000 7fb19eae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.451+0000 7fb19eae6640 1 -- 172.21.15.32:0/2417834589 >> v1:172.21.15.32:6790/0 conn(0x7fb1980b4d60 legacy=0x7fb19813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.451+0000 7fb19eae6640 1 -- 172.21.15.32:0/2417834589 shutdown_connections 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.451+0000 7fb19eae6640 1 -- 172.21.15.32:0/2417834589 >> 172.21.15.32:0/2417834589 conn(0x7fb1980617a0 msgr2=0x7fb198061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.451+0000 7fb19eae6640 1 -- 172.21.15.32:0/2417834589 shutdown_connections 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.451+0000 7fb19eae6640 1 -- 172.21.15.32:0/2417834589 wait complete. 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 10 monclient: build_initial_monmap 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 10 monclient: monmap: 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:01.454 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 1 librados: starting msgr at 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 1 librados: starting objecter 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 1 Processor -- start 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 1 -- start start 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 1 librados: setting wanted keys 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 1 librados: calling monclient init 2024-03-25T18:45:01.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.452+0000 7fb19eae6640 10 monclient: init 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.453+0000 7fb19eae6640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.453+0000 7fb19eae6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.453+0000 7fb19eae6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1980b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.453+0000 7fb19eae6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1980b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.453+0000 7fb19eae6640 10 monclient(hunting): picked mon.noname-b con 0x7fb19814be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.454+0000 7fb19eae6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb198138010 con 0x7fb1980b1730 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.454+0000 7fb19eae6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1981382f0 con 0x7fb19814be90 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.454+0000 7fb19eae6640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb198136d90 con 0x7fb1980b4d60 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.454+0000 7fb19eae6640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:01.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.454+0000 7fb19eae6640 10 monclient(hunting): authenticate will time out at 6407.643618s 2024-03-25T18:45:01.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.457+0000 7fb197fff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fb1980b4d60 0x7fb198143900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:36586/0 (socket says 172.21.15.32:36586) 2024-03-25T18:45:01.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.457+0000 7fb197fff640 1 -- 172.21.15.32:0/2366545908 learned_addr learned my addr 172.21.15.32:0/2366545908 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:01.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.457+0000 7fb1957fa640 10 client.?.objecter ms_handle_connect 0x7fb1980b4d60 2024-03-25T18:45:01.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.457+0000 7fb1957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:01.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.458+0000 7fb1957fa640 10 client.?.objecter ms_handle_connect 0x7fb1980b1730 2024-03-25T18:45:01.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.458+0000 7fb1957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:01.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.463+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2791009515 0 0) 0x7fb198138010 con 0x7fb1980b1730 2024-03-25T18:45:01.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.463+0000 7fb1957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:01.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.463+0000 7fb1957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:01.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.463+0000 7fb1957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:01.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.463+0000 7fb1957fa640 10 monclient(hunting): my global_id is 4491 2024-03-25T18:45:01.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.463+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb17c004eb0 con 0x7fb1980b1730 2024-03-25T18:45:01.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.463+0000 7fb1957fa640 10 client.?.objecter ms_handle_connect 0x7fb19814be90 2024-03-25T18:45:01.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.463+0000 7fb1957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:01.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.469+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2562680126 0 0) 0x7fb198136d90 con 0x7fb1980b4d60 2024-03-25T18:45:01.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.469+0000 7fb1957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:01.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.469+0000 7fb1957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:01.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.469+0000 7fb1957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:01.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.469+0000 7fb1957fa640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:45:01.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.469+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb198138010 con 0x7fb1980b4d60 2024-03-25T18:45:01.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.471+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 481466845 0 0) 0x7fb17c004eb0 con 0x7fb1980b1730 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.471+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb198136d90 con 0x7fb1980b1730 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.471+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb18c001b00 con 0x7fb1980b1730 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.471+0000 7fb1957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.471+0000 7fb1957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.471+0000 7fb1957fa640 10 monclient(hunting): dump: 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:01.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.471+0000 7fb1957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:45:01.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.477+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635776436 0 0) 0x7fb1981382f0 con 0x7fb19814be90 2024-03-25T18:45:01.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.477+0000 7fb1957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:01.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.477+0000 7fb1957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:01.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.477+0000 7fb1957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:01.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.477+0000 7fb1957fa640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:45:01.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.477+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb17c004eb0 con 0x7fb19814be90 2024-03-25T18:45:01.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 35584976 0 0) 0x7fb198136d90 con 0x7fb1980b1730 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 >> v1:172.21.15.32:6790/0 conn(0x7fb1980b4d60 legacy=0x7fb198143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 >> v1:172.21.15.136:6789/0 conn(0x7fb19814be90 legacy=0x7fb1981427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 1 monclient: found mon.a 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb198144010 con 0x7fb1980b1730 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 10 monclient: _finish_auth 0 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb1957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 5 monclient: authenticate success, global_id 4491 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 10 monclient: _renew_subs 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:01.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1981372b0 con 0x7fb1980b1730 2024-03-25T18:45:01.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 10 client.4491.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:45:01.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 10 monclient: _renew_subs 2024-03-25T18:45:01.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:01.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1981369c0 con 0x7fb1980b1730 2024-03-25T18:45:01.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.479+0000 7fb19eae6640 1 librados: init done 2024-03-25T18:45:01.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.480+0000 7fb19eae6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:01.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.480+0000 7fb19eae6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:01.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.480+0000 7fb19eae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:01.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.480+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb198136f20 con 0x7fb1980b1730 2024-03-25T18:45:01.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.786+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb18c001220 con 0x7fb1980b1730 2024-03-25T18:45:01.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.786+0000 7fb1957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.786+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb18c004c30 con 0x7fb1980b1730 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.786+0000 7fb1957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.786+0000 7fb1957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.786+0000 7fb1957fa640 10 monclient: dump: 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:01.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.786+0000 7fb196ffd640 4 set_mon_vals no callback set 2024-03-25T18:45:01.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.787+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fb18c01ac00 con 0x7fb1980b1730 2024-03-25T18:45:01.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.787+0000 7fb197fff640 10 monclient: get_auth_request con 0x7fb17c06ddb0 auth_method 0 2024-03-25T18:45:01.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.788+0000 7fb1957fa640 10 client.4491.objecter ms_handle_connect 0x7fb17c06ddb0 2024-03-25T18:45:01.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.792+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fb18c054870 con 0x7fb1980b1730 2024-03-25T18:45:01.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.793+0000 7fb1957fa640 10 client.4491.objecter ms_dispatch 0x7fb198000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:45:01.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.793+0000 7fb1957fa640 3 client.4491.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:45:01.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.793+0000 7fb1957fa640 3 client.4491.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:45:01.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.793+0000 7fb1957fa640 20 client.4491.objecter dump_active .. 0 homeless 2024-03-25T18:45:01.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.793+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb18c0501b0 con 0x7fb1980b1730 2024-03-25T18:45:01.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.793+0000 7fb1957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:01.794 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:01.793+0000 7fb1957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:45:02.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.044+0000 7fb19eae6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:02.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.044+0000 7fb19eae6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:02.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.044+0000 7fb19eae6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:02.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.044+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb1981441a0 con 0x7fb1980b1730 2024-03-25T18:45:02.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.073+0000 7fb1957fa640 1 -- 172.21.15.32:0/2366545908 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fb18c0522b0 con 0x7fb1980b1730 2024-03-25T18:45:02.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.073+0000 7fb1957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:02.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.073+0000 7fb1957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:45:02.075 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:45:02.075 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:45:02.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.075+0000 7fb19eae6640 10 librados: watch_flush enter 2024-03-25T18:45:02.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.075+0000 7fb19eae6640 10 librados: watch_flush exit 2024-03-25T18:45:02.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.075+0000 7fb19eae6640 20 client.4491.objecter shutdown clearing up homeless session... 2024-03-25T18:45:02.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.075+0000 7fb19eae6640 10 client.4491.objecter successfully canceled tick 2024-03-25T18:45:02.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.075+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 >> v1:172.21.15.136:6800/1141502808 conn(0x7fb17c06ddb0 legacy=0x7fb17c070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:02.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.075+0000 7fb19eae6640 10 monclient: shutdown 2024-03-25T18:45:02.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.075+0000 7fb19eae6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:02.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.075+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 >> v1:172.21.15.32:6789/0 conn(0x7fb1980b1730 legacy=0x7fb198135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:02.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.076+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 shutdown_connections 2024-03-25T18:45:02.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.076+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 >> 172.21.15.32:0/2366545908 conn(0x7fb1980617a0 msgr2=0x7fb1980b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:02.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.076+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 shutdown_connections 2024-03-25T18:45:02.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.076+0000 7fb19eae6640 1 -- 172.21.15.32:0/2366545908 wait complete. 2024-03-25T18:45:02.078 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:02.076+0000 7fb19eae6640 1 librados: shutdown 2024-03-25T18:45:02.089 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:45:07.971 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:20.299 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.015 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:20.331 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.016 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:20.333 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.023 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:20.351 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.053 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:20.381 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.065 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:20.382 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.066 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:20.394 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.083 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:20.411 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.116 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:20.444 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:08.895 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:45:09.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.008+0000 7f125efd4640 10 monclient: get_monmap_and_config 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.008+0000 7f125efd4640 10 monclient: build_initial_monmap 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.008+0000 7f125efd4640 10 monclient: monmap: 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:45:09.010202+0000 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:45:09.010202+0000 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:45:09.010 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.009+0000 7f125efd4640 1 Processor -- start 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.009+0000 7f125efd4640 1 -- start start 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.009+0000 7f125efd4640 10 monclient: init 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 10 monclient(hunting): picked mon.noname-b con 0x7f12580a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 10 monclient(hunting): picked mon.noname-c con 0x7f125813f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 10 monclient(hunting): picked mon.noname-a con 0x7f12581430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1258142520 con 0x7f12581430c0 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12580b28a0 con 0x7f125813f710 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12580b2a30 con 0x7f12580a98f0 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:09.012 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.010+0000 7f125efd4640 10 monclient(hunting): authenticate will time out at 6415.200264s 2024-03-25T18:45:09.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.011+0000 7f1257fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f125813f710 0x7f12581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:45266/0 (socket says 172.21.15.32:45266) 2024-03-25T18:45:09.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.011+0000 7f1257fff640 1 -- 172.21.15.32:0/325434361 learned_addr learned my addr 172.21.15.32:0/325434361 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:09.103 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.102+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 12326223 0 0) 0x7f1258142520 con 0x7f12581430c0 2024-03-25T18:45:09.103 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.102+0000 7f12577fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:09.103 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.102+0000 7f12577fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:09.103 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.102+0000 7f12577fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:09.103 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.102+0000 7f12577fe640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:45:09.104 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.102+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1248003520 con 0x7f12581430c0 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4242721969 0 0) 0x7f1248003520 con 0x7f12581430c0 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 >> v1:172.21.15.32:6790/0 conn(0x7f12580a98f0 legacy=0x7f12580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 >> v1:172.21.15.136:6789/0 conn(0x7f125813f710 legacy=0x7f12581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 10 monclient: _finish_hunting 0 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 1 monclient: found mon.noname-a 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12580b44c0 con 0x7f12581430c0 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 10 monclient: _finish_auth 0 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 10 monclient: _check_auth_tickets 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1250001a10 con 0x7f12581430c0 2024-03-25T18:45:09.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f125efd4640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f125efd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f12577fe640 10 monclient: dump: 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:09.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.111+0000 7f125efd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1250001b70 con 0x7f12581430c0 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f12577fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f12577fe640 1 -- 172.21.15.32:0/325434361 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1250001f80 con 0x7f12581430c0 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 10 monclient: get_monmap_and_config success 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 4 set_mon_vals no callback set 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f12577fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f12577fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f12577fe640 10 monclient: dump: 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:09.118 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:09.119 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:09.119 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:09.119 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:09.119 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:09.119 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:09.119 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 10 monclient: shutdown 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 1 -- 172.21.15.32:0/325434361 >> v1:172.21.15.32:6789/0 conn(0x7f12581430c0 legacy=0x7f125814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 1 -- 172.21.15.32:0/325434361 shutdown_connections 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 1 -- 172.21.15.32:0/325434361 >> 172.21.15.32:0/325434361 conn(0x7f12580b7ea0 msgr2=0x7f12580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 1 -- 172.21.15.32:0/325434361 shutdown_connections 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.117+0000 7f125efd4640 1 -- 172.21.15.32:0/325434361 wait complete. 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.118+0000 7f125efd4640 10 monclient: build_initial_monmap 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.118+0000 7f125efd4640 10 monclient: monmap: 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:09.121 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.118+0000 7f125efd4640 1 librados: starting msgr at 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.118+0000 7f125efd4640 1 librados: starting objecter 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 1 Processor -- start 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 1 -- start start 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 1 librados: setting wanted keys 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 1 librados: calling monclient init 2024-03-25T18:45:09.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 10 monclient: init 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 10 monclient(hunting): picked mon.noname-b con 0x7f12580a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 10 monclient(hunting): picked mon.noname-c con 0x7f125813f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 10 monclient(hunting): picked mon.noname-a con 0x7f12581430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.119+0000 7f125efd4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1258137f10 con 0x7f12581430c0 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.120+0000 7f125efd4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12581dc070 con 0x7f12580a98f0 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.120+0000 7f125efd4640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1258137100 con 0x7f125813f710 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.120+0000 7f125efd4640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:09.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.120+0000 7f125efd4640 10 monclient(hunting): authenticate will time out at 6415.309575s 2024-03-25T18:45:09.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.128+0000 7f125d571640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f12581430c0 0x7f12581db960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43914/0 (socket says 172.21.15.32:43914) 2024-03-25T18:45:09.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.128+0000 7f125d571640 1 -- 172.21.15.32:0/303422716 learned_addr learned my addr 172.21.15.32:0/303422716 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:09.130 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.129+0000 7f12557fa640 10 client.?.objecter ms_handle_connect 0x7f12581430c0 2024-03-25T18:45:09.131 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.129+0000 7f12557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:09.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.138+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3095001495 0 0) 0x7f1258137f10 con 0x7f12581430c0 2024-03-25T18:45:09.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.138+0000 7f12557fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:09.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.138+0000 7f12557fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:09.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.138+0000 7f12557fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:09.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.138+0000 7f12557fa640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:45:09.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.138+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f123c004eb0 con 0x7f12581430c0 2024-03-25T18:45:09.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.144+0000 7f12557fa640 10 client.?.objecter ms_handle_connect 0x7f12580a98f0 2024-03-25T18:45:09.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.144+0000 7f12557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:09.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.156+0000 7f12557fa640 10 client.?.objecter ms_handle_connect 0x7f125813f710 2024-03-25T18:45:09.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.156+0000 7f12557fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:09.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.168+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1643495209 0 0) 0x7f123c004eb0 con 0x7f12581430c0 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.168+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1258137f10 con 0x7f12581430c0 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.168+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1250001fc0 con 0x7f12581430c0 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.168+0000 7f12557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.168+0000 7f12557fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.168+0000 7f12557fa640 10 monclient(hunting): dump: 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:09.170 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:09.171 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.168+0000 7f12557fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:45:09.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2103634216 0 0) 0x7f1258137f10 con 0x7f12581430c0 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 >> v1:172.21.15.32:6790/0 conn(0x7f125813f710 legacy=0x7f12581db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 >> v1:172.21.15.136:6789/0 conn(0x7f12580a98f0 legacy=0x7f1258137800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 10 monclient: _finish_hunting 0 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 1 monclient: found mon.a 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12581373e0 con 0x7f12581430c0 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 10 monclient: _finish_auth 0 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 10 monclient: _check_auth_tickets 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f12557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 5 monclient: authenticate success, global_id 4497 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 10 monclient: _renew_subs 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:09.229 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12581358e0 con 0x7f12581430c0 2024-03-25T18:45:09.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 10 client.4497.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:45:09.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 10 monclient: _renew_subs 2024-03-25T18:45:09.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:09.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1258136150 con 0x7f12581430c0 2024-03-25T18:45:09.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.227+0000 7f125efd4640 1 librados: init done 2024-03-25T18:45:09.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.228+0000 7f125efd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:09.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.228+0000 7f125efd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:09.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.228+0000 7f125efd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:09.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.228+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12580627b0 con 0x7f12581430c0 2024-03-25T18:45:09.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.552+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1250003c20 con 0x7f12581430c0 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.552+0000 7f12557fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.552+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1250004030 con 0x7f12581430c0 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.552+0000 7f12557fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.552+0000 7f12557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.552+0000 7f12557fa640 10 monclient: dump: 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:09.554 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:09.555 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:09.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.552+0000 7f1256ffd640 4 set_mon_vals no callback set 2024-03-25T18:45:09.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.553+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f1250004250 con 0x7f12581430c0 2024-03-25T18:45:09.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.554+0000 7f125cd70640 10 monclient: get_auth_request con 0x7f123c06dd00 auth_method 0 2024-03-25T18:45:09.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.554+0000 7f12557fa640 10 client.4497.objecter ms_handle_connect 0x7f123c06dd00 2024-03-25T18:45:09.563 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.562+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f1250054c60 con 0x7f12581430c0 2024-03-25T18:45:09.563 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.562+0000 7f12557fa640 10 client.4497.objecter ms_dispatch 0x7f1258000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:45:09.563 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.562+0000 7f12557fa640 3 client.4497.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:45:09.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.562+0000 7f12557fa640 3 client.4497.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:45:09.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.562+0000 7f12557fa640 20 client.4497.objecter dump_active .. 0 homeless 2024-03-25T18:45:09.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.562+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1250050770 con 0x7f12581430c0 2024-03-25T18:45:09.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.562+0000 7f12557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:09.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.562+0000 7f12557fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:45:09.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.810+0000 7f125efd4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:09.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.810+0000 7f125efd4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:09.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.810+0000 7f125efd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:09.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.810+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f125813e250 con 0x7f12581430c0 2024-03-25T18:45:09.847 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.846+0000 7f12557fa640 1 -- 172.21.15.32:0/303422716 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f12500526a0 con 0x7f12581430c0 2024-03-25T18:45:09.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.846+0000 7f12557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:09.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.846+0000 7f12557fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:45:09.848 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:45:09.848 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:45:09.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 10 librados: watch_flush enter 2024-03-25T18:45:09.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 10 librados: watch_flush exit 2024-03-25T18:45:09.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 20 client.4497.objecter shutdown clearing up homeless session... 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 10 client.4497.objecter successfully canceled tick 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 >> v1:172.21.15.136:6800/1141502808 conn(0x7f123c06dd00 legacy=0x7f123c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 10 monclient: shutdown 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 >> v1:172.21.15.32:6789/0 conn(0x7f12581430c0 legacy=0x7f12581db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 shutdown_connections 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 >> 172.21.15.32:0/303422716 conn(0x7f12580b7ea0 msgr2=0x7f125814ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 shutdown_connections 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 1 -- 172.21.15.32:0/303422716 wait complete. 2024-03-25T18:45:09.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:09.849+0000 7f125efd4640 1 librados: shutdown 2024-03-25T18:45:09.864 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:45:16.671 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:45:16.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.787+0000 7f68487c2640 10 monclient: get_monmap_and_config 2024-03-25T18:45:16.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.787+0000 7f68487c2640 10 monclient: build_initial_monmap 2024-03-25T18:45:16.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.787+0000 7f68487c2640 10 monclient: monmap: 2024-03-25T18:45:16.789 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:45:16.789072+0000 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:45:16.789072+0000 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:16.790 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.788+0000 7f68487c2640 1 Processor -- start 2024-03-25T18:45:16.791 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.788+0000 7f68487c2640 1 -- start start 2024-03-25T18:45:16.791 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.788+0000 7f68487c2640 10 monclient: init 2024-03-25T18:45:16.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:16.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:45:16.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 10 monclient(hunting): picked mon.noname-c con 0x7f68400a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:16.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 10 monclient(hunting): picked mon.noname-b con 0x7f684013f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:16.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 10 monclient(hunting): picked mon.noname-a con 0x7f68401430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:16.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6840142520 con 0x7f68401430c0 2024-03-25T18:45:16.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68400b28a0 con 0x7f68400a98f0 2024-03-25T18:45:16.793 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68400b2a30 con 0x7f684013f710 2024-03-25T18:45:16.793 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:16.793 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.789+0000 7f68487c2640 10 monclient(hunting): authenticate will time out at 6422.978819s 2024-03-25T18:45:16.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.795+0000 7f6846d5f640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f68401430c0 0x7f684014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50660/0 (socket says 172.21.15.32:50660) 2024-03-25T18:45:16.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.795+0000 7f6846d5f640 1 -- 172.21.15.32:0/588359783 learned_addr learned my addr 172.21.15.32:0/588359783 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:16.813 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.812+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122456941 0 0) 0x7f68400b28a0 con 0x7f68400a98f0 2024-03-25T18:45:16.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.812+0000 7f684555c640 10 monclient(hunting): _negotiate 2024-03-25T18:45:16.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.812+0000 7f684555c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:16.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.812+0000 7f684555c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:16.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.812+0000 7f684555c640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:45:16.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.812+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f682c003520 con 0x7f68400a98f0 2024-03-25T18:45:16.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.815+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314769569 0 0) 0x7f6840142520 con 0x7f68401430c0 2024-03-25T18:45:16.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.815+0000 7f684555c640 10 monclient(hunting): _negotiate 2024-03-25T18:45:16.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.815+0000 7f684555c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:16.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.815+0000 7f684555c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:16.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.815+0000 7f684555c640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:45:16.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.815+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68400b28a0 con 0x7f68401430c0 2024-03-25T18:45:16.819 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.818+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358566457 0 0) 0x7f68400b2a30 con 0x7f684013f710 2024-03-25T18:45:16.819 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.818+0000 7f684555c640 10 monclient(hunting): _negotiate 2024-03-25T18:45:16.819 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.818+0000 7f684555c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:16.819 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.818+0000 7f684555c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:16.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.818+0000 7f684555c640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:45:16.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.818+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6840142520 con 0x7f684013f710 2024-03-25T18:45:16.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4082319614 0 0) 0x7f682c003520 con 0x7f68400a98f0 2024-03-25T18:45:16.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 >> v1:172.21.15.32:6790/0 conn(0x7f684013f710 legacy=0x7f68401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:16.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 >> v1:172.21.15.32:6789/0 conn(0x7f68401430c0 legacy=0x7f684014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:16.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 10 monclient: _finish_hunting 0 2024-03-25T18:45:16.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 1 monclient: found mon.noname-c 2024-03-25T18:45:16.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:16.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68400b44c0 con 0x7f68400a98f0 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 10 monclient: _finish_auth 0 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 10 monclient: _check_auth_tickets 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f683c001690 con 0x7f68400a98f0 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f684555c640 10 monclient: dump: 2024-03-25T18:45:16.833 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:16.834 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:16.834 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:16.834 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:16.834 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:16.834 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:16.834 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:16.834 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:16.834 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:16.835 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:16.835 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f68487c2640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:45:16.835 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.830+0000 7f68487c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f683c0017f0 con 0x7f68400a98f0 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f684555c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f684555c640 1 -- 172.21.15.32:0/588359783 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f683c001c00 con 0x7f68400a98f0 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f684555c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f684555c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f684555c640 10 monclient: dump: 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:16.854 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:16.855 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:16.855 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:16.855 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:16.855 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:16.855 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:16.855 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:16.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f68487c2640 10 monclient: get_monmap_and_config success 2024-03-25T18:45:16.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f68487c2640 4 set_mon_vals no callback set 2024-03-25T18:45:16.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f68487c2640 10 monclient: shutdown 2024-03-25T18:45:16.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f68487c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:16.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f68487c2640 1 -- 172.21.15.32:0/588359783 >> v1:172.21.15.136:6789/0 conn(0x7f68400a98f0 legacy=0x7f68400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:16.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f68487c2640 1 -- 172.21.15.32:0/588359783 shutdown_connections 2024-03-25T18:45:16.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.853+0000 7f68487c2640 1 -- 172.21.15.32:0/588359783 >> 172.21.15.32:0/588359783 conn(0x7f68400b7ea0 msgr2=0x7f68400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:16.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.854+0000 7f68487c2640 1 -- 172.21.15.32:0/588359783 shutdown_connections 2024-03-25T18:45:16.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.854+0000 7f68487c2640 1 -- 172.21.15.32:0/588359783 wait complete. 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.854+0000 7f68487c2640 10 monclient: build_initial_monmap 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.854+0000 7f68487c2640 10 monclient: monmap: 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.855+0000 7f68487c2640 1 librados: starting msgr at 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.855+0000 7f68487c2640 1 librados: starting objecter 2024-03-25T18:45:16.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.855+0000 7f68487c2640 1 Processor -- start 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.855+0000 7f68487c2640 1 -- start start 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.855+0000 7f68487c2640 1 librados: setting wanted keys 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.855+0000 7f68487c2640 1 librados: calling monclient init 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.855+0000 7f68487c2640 10 monclient: init 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.855+0000 7f68487c2640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 10 monclient(hunting): picked mon.noname-c con 0x7f68400a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 10 monclient(hunting): picked mon.noname-b con 0x7f684013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 10 monclient(hunting): picked mon.noname-a con 0x7f68401430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6840135730 con 0x7f68401430c0 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68401358c0 con 0x7f684013f710 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6840135a50 con 0x7f68400a98f0 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:16.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.856+0000 7f68487c2640 10 monclient(hunting): authenticate will time out at 6423.045720s 2024-03-25T18:45:16.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.867+0000 7f6846d5f640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f68401430c0 0x7f68401d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:50736/0 (socket says 172.21.15.32:50736) 2024-03-25T18:45:16.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.867+0000 7f6846d5f640 1 -- 172.21.15.32:0/1240055090 learned_addr learned my addr 172.21.15.32:0/1240055090 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:16.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.867+0000 7f6836ffd640 10 client.?.objecter ms_handle_connect 0x7f68401430c0 2024-03-25T18:45:16.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.867+0000 7f6836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:16.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.869+0000 7f6836ffd640 10 client.?.objecter ms_handle_connect 0x7f684013f710 2024-03-25T18:45:16.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.869+0000 7f6836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:16.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.877+0000 7f6836ffd640 10 client.?.objecter ms_handle_connect 0x7f68400a98f0 2024-03-25T18:45:16.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.877+0000 7f6836ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:16.884 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.882+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116645802 0 0) 0x7f68401358c0 con 0x7f684013f710 2024-03-25T18:45:16.884 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.882+0000 7f6836ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:45:16.884 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.882+0000 7f6836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:16.884 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.882+0000 7f6836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:16.884 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.882+0000 7f6836ffd640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:45:16.884 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.882+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6824004eb0 con 0x7f684013f710 2024-03-25T18:45:16.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.887+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 126620934 0 0) 0x7f6840135730 con 0x7f68401430c0 2024-03-25T18:45:16.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.887+0000 7f6836ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:45:16.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.887+0000 7f6836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:16.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.887+0000 7f6836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:16.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.887+0000 7f6836ffd640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:45:16.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.887+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68401358c0 con 0x7f68401430c0 2024-03-25T18:45:16.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.893+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2083713033 0 0) 0x7f6840135a50 con 0x7f68400a98f0 2024-03-25T18:45:16.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.893+0000 7f6836ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:45:16.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.893+0000 7f6836ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:16.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.893+0000 7f6836ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:16.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.893+0000 7f6836ffd640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:45:16.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.893+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6840135730 con 0x7f68400a98f0 2024-03-25T18:45:16.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.896+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3983502825 0 0) 0x7f6824004eb0 con 0x7f684013f710 2024-03-25T18:45:16.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.896+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6840135a50 con 0x7f684013f710 2024-03-25T18:45:16.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.896+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f68300018f0 con 0x7f684013f710 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.896+0000 7f6836ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.896+0000 7f6836ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.896+0000 7f6836ffd640 10 monclient(hunting): dump: 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:16.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.896+0000 7f6836ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:45:16.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3962139136 0 0) 0x7f6840135a50 con 0x7f684013f710 2024-03-25T18:45:16.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 >> v1:172.21.15.32:6790/0 conn(0x7f68400a98f0 legacy=0x7f68401dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:16.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 >> v1:172.21.15.32:6789/0 conn(0x7f68401430c0 legacy=0x7f68401d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 1 monclient: found mon.b 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68401d5000 con 0x7f684013f710 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 10 monclient: _finish_auth 0 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.902+0000 7f6836ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 5 monclient: authenticate success, global_id 4486 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 10 monclient: _renew_subs 2024-03-25T18:45:16.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:16.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68401d52b0 con 0x7f684013f710 2024-03-25T18:45:16.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 10 client.4486.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:45:16.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 10 monclient: _renew_subs 2024-03-25T18:45:16.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:16.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f684013c900 con 0x7f684013f710 2024-03-25T18:45:16.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 1 librados: init done 2024-03-25T18:45:16.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:16.907 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:16.907 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:16.907 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:16.903+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68401371c0 con 0x7f684013f710 2024-03-25T18:45:17.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.201+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6830003610 con 0x7f684013f710 2024-03-25T18:45:17.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.201+0000 7f6836ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:17.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.201+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6830003a20 con 0x7f684013f710 2024-03-25T18:45:17.203 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.201+0000 7f6836ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:17.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.201+0000 7f6836ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:45:17.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.201+0000 7f6836ffd640 10 monclient: dump: 2024-03-25T18:45:17.204 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:17.204 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:17.204 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:17.204 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:17.205 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:17.205 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:17.205 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:17.205 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:17.205 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:17.205 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:17.206 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.201+0000 7f6844d5b640 4 set_mon_vals no callback set 2024-03-25T18:45:17.206 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.202+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f6830003c40 con 0x7f684013f710 2024-03-25T18:45:17.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.202+0000 7f684655e640 10 monclient: get_auth_request con 0x7f682406ddf0 auth_method 0 2024-03-25T18:45:17.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.203+0000 7f6836ffd640 10 client.4486.objecter ms_handle_connect 0x7f682406ddf0 2024-03-25T18:45:17.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.211+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f6830053650 con 0x7f684013f710 2024-03-25T18:45:17.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.211+0000 7f6836ffd640 10 client.4486.objecter ms_dispatch 0x7f6840000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:45:17.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.211+0000 7f6836ffd640 3 client.4486.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:45:17.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.211+0000 7f6836ffd640 3 client.4486.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:45:17.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.212+0000 7f6836ffd640 20 client.4486.objecter dump_active .. 0 homeless 2024-03-25T18:45:17.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.212+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f683004f160 con 0x7f684013f710 2024-03-25T18:45:17.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.212+0000 7f6836ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:17.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.212+0000 7f6836ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:45:17.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.463+0000 7f68487c2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:17.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.463+0000 7f68487c2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:17.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.463+0000 7f68487c2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:17.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.463+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f68400627b0 con 0x7f684013f710 2024-03-25T18:45:17.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.487+0000 7f6836ffd640 1 -- 172.21.15.32:0/1240055090 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f6830051090 con 0x7f684013f710 2024-03-25T18:45:17.506 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.487+0000 7f6836ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:17.507 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.487+0000 7f6836ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:45:17.507 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:45:17.508 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:45:17.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.489+0000 7f68487c2640 10 librados: watch_flush enter 2024-03-25T18:45:17.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 10 librados: watch_flush exit 2024-03-25T18:45:17.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 20 client.4486.objecter shutdown clearing up homeless session... 2024-03-25T18:45:17.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 10 client.4486.objecter successfully canceled tick 2024-03-25T18:45:17.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 >> v1:172.21.15.136:6800/1141502808 conn(0x7f682406ddf0 legacy=0x7f68240702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:17.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 10 monclient: shutdown 2024-03-25T18:45:17.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:17.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 >> v1:172.21.15.136:6789/0 conn(0x7f684013f710 legacy=0x7f68401d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:17.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 shutdown_connections 2024-03-25T18:45:17.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 >> 172.21.15.32:0/1240055090 conn(0x7f68400b7ea0 msgr2=0x7f684014ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:17.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 shutdown_connections 2024-03-25T18:45:17.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 1 -- 172.21.15.32:0/1240055090 wait complete. 2024-03-25T18:45:17.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:17.490+0000 7f68487c2640 1 librados: shutdown 2024-03-25T18:45:17.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:45:24.320 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:45:24.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.429+0000 7f5886dd0640 10 monclient: get_monmap_and_config 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.429+0000 7f5886dd0640 10 monclient: build_initial_monmap 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.429+0000 7f5886dd0640 10 monclient: monmap: 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:45:24.430991+0000 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:45:24.430991+0000 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:45:24.431 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:24.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.430+0000 7f5886dd0640 1 Processor -- start 2024-03-25T18:45:24.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.430+0000 7f5886dd0640 1 -- start start 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.430+0000 7f5886dd0640 10 monclient: init 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.430+0000 7f5886dd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.430+0000 7f5886dd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.430+0000 7f5886dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f58800b10d0 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.430+0000 7f5886dd0640 10 monclient(hunting): picked mon.noname-c con 0x7f58800b4360 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.430+0000 7f5886dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f58800a9470 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.431+0000 7f5886dd0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800a9ab0 con 0x7f58800b10d0 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.431+0000 7f5886dd0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800b2400 con 0x7f58800b4360 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.431+0000 7f5886dd0640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800aa2a0 con 0x7f58800a9470 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.431+0000 7f5886dd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:24.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.431+0000 7f5886dd0640 10 monclient(hunting): authenticate will time out at 6430.620564s 2024-03-25T18:45:24.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.439+0000 7f588536d640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f58800a9470 0x7f588014dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:45038/0 (socket says 172.21.15.32:45038) 2024-03-25T18:45:24.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.439+0000 7f588536d640 1 -- 172.21.15.32:0/220570564 learned_addr learned my addr 172.21.15.32:0/220570564 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:24.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.449+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 148372979 0 0) 0x7f58800a9ab0 con 0x7f58800b10d0 2024-03-25T18:45:24.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.449+0000 7f587f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:24.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.449+0000 7f587f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:24.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.449+0000 7f587f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:24.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.449+0000 7f587f7fe640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:45:24.451 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.449+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5870003520 con 0x7f58800b10d0 2024-03-25T18:45:24.461 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.460+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1096035867 0 0) 0x7f58800b2400 con 0x7f58800b4360 2024-03-25T18:45:24.461 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.460+0000 7f587f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:24.461 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.460+0000 7f587f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:24.462 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.460+0000 7f587f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:24.462 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.460+0000 7f587f7fe640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:45:24.462 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.460+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58800a9ab0 con 0x7f58800b4360 2024-03-25T18:45:24.463 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.462+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 877807414 0 0) 0x7f58800aa2a0 con 0x7f58800a9470 2024-03-25T18:45:24.463 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.462+0000 7f587f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:24.463 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.462+0000 7f587f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:24.463 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.462+0000 7f587f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:24.463 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.462+0000 7f587f7fe640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:45:24.463 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.462+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58800b2400 con 0x7f58800a9470 2024-03-25T18:45:24.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1071963353 0 0) 0x7f5870003520 con 0x7f58800b10d0 2024-03-25T18:45:24.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 >> v1:172.21.15.32:6790/0 conn(0x7f58800a9470 legacy=0x7f588014dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:24.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 >> v1:172.21.15.136:6789/0 conn(0x7f58800b4360 legacy=0x7f58800b1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:24.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:45:24.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58800b14b0 con 0x7f58800b10d0 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5874001690 con 0x7f58800b10d0 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f5886dd0640 5 monclient: authenticate success, global_id 4506 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f587f7fe640 10 monclient: dump: 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.468 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:24.469 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:24.469 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:24.469 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:24.469 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:24.469 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:24.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.466+0000 7f5886dd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.476+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58740017f0 con 0x7f58800b10d0 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.476+0000 7f587f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.476+0000 7f587f7fe640 1 -- 172.21.15.32:0/220570564 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5874001c00 con 0x7f58800b10d0 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.476+0000 7f5886dd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.476+0000 7f5886dd0640 4 set_mon_vals no callback set 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.476+0000 7f587f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.476+0000 7f587f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.476+0000 7f587f7fe640 10 monclient: dump: 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.478 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.479 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:24.479 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:24.479 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:24.479 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:24.479 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:24.479 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:24.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.477+0000 7f5886dd0640 10 monclient: shutdown 2024-03-25T18:45:24.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.477+0000 7f5886dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:24.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.477+0000 7f5886dd0640 1 -- 172.21.15.32:0/220570564 >> v1:172.21.15.32:6789/0 conn(0x7f58800b10d0 legacy=0x7f58800b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:24.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.477+0000 7f5886dd0640 1 -- 172.21.15.32:0/220570564 shutdown_connections 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.477+0000 7f5886dd0640 1 -- 172.21.15.32:0/220570564 >> 172.21.15.32:0/220570564 conn(0x7f58800617a0 msgr2=0x7f5880061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.478+0000 7f5886dd0640 1 -- 172.21.15.32:0/220570564 shutdown_connections 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.478+0000 7f5886dd0640 1 -- 172.21.15.32:0/220570564 wait complete. 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.478+0000 7f5886dd0640 10 monclient: build_initial_monmap 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.478+0000 7f5886dd0640 10 monclient: monmap: 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:24.482 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.479+0000 7f5886dd0640 1 librados: starting msgr at 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.479+0000 7f5886dd0640 1 librados: starting objecter 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.479+0000 7f5886dd0640 1 Processor -- start 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.479+0000 7f5886dd0640 1 -- start start 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.479+0000 7f5886dd0640 1 librados: setting wanted keys 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.479+0000 7f5886dd0640 1 librados: calling monclient init 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.479+0000 7f5886dd0640 10 monclient: init 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 10 monclient(hunting): picked mon.noname-a con 0x7f58800a9470 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 10 monclient(hunting): picked mon.noname-b con 0x7f58800b10d0 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 10 monclient(hunting): picked mon.noname-c con 0x7f58800b4360 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5880133de0 con 0x7f58800a9470 2024-03-25T18:45:24.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588013db10 con 0x7f58800b10d0 2024-03-25T18:45:24.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f588013cd00 con 0x7f58800b4360 2024-03-25T18:45:24.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:24.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.480+0000 7f5886dd0640 10 monclient(hunting): authenticate will time out at 6430.669805s 2024-03-25T18:45:24.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.484+0000 7f588536d640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f58800b4360 0x7f58801e2220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:45060/0 (socket says 172.21.15.32:45060) 2024-03-25T18:45:24.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.484+0000 7f588536d640 1 -- 172.21.15.32:0/1217085402 learned_addr learned my addr 172.21.15.32:0/1217085402 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:24.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.485+0000 7f587d7fa640 10 client.?.objecter ms_handle_connect 0x7f58800b4360 2024-03-25T18:45:24.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.485+0000 7f587d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:24.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.494+0000 7f587d7fa640 10 client.?.objecter ms_handle_connect 0x7f58800a9470 2024-03-25T18:45:24.495 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.494+0000 7f587d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:24.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.495+0000 7f587d7fa640 10 client.?.objecter ms_handle_connect 0x7f58800b10d0 2024-03-25T18:45:24.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.495+0000 7f587d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:24.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.501+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48010549 0 0) 0x7f588013db10 con 0x7f58800b10d0 2024-03-25T18:45:24.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.501+0000 7f587d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:24.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.501+0000 7f587d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:24.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.501+0000 7f587d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:24.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.501+0000 7f587d7fa640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:45:24.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.501+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5864004eb0 con 0x7f58800b10d0 2024-03-25T18:45:24.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.504+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103497878 0 0) 0x7f5880133de0 con 0x7f58800a9470 2024-03-25T18:45:24.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.504+0000 7f587d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:24.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.504+0000 7f587d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:24.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.504+0000 7f587d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:24.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.504+0000 7f587d7fa640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:45:24.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.504+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f588013db10 con 0x7f58800a9470 2024-03-25T18:45:24.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.508+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264310465 0 0) 0x7f588013cd00 con 0x7f58800b4360 2024-03-25T18:45:24.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.508+0000 7f587d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:24.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.508+0000 7f587d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:24.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.508+0000 7f587d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:24.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.508+0000 7f587d7fa640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:45:24.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.508+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880133de0 con 0x7f58800b4360 2024-03-25T18:45:24.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.519+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2196024058 0 0) 0x7f588013db10 con 0x7f58800a9470 2024-03-25T18:45:24.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.519+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f588013cd00 con 0x7f58800a9470 2024-03-25T18:45:24.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.519+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5874001b00 con 0x7f58800a9470 2024-03-25T18:45:24.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.519+0000 7f587d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:24.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.519+0000 7f587d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.519+0000 7f587d7fa640 10 monclient(hunting): dump: 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:24.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.519+0000 7f587d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:45:24.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.524+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2453828544 0 0) 0x7f5880133de0 con 0x7f58800b4360 2024-03-25T18:45:24.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.524+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f588013db10 con 0x7f58800b4360 2024-03-25T18:45:24.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.524+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5878001bd0 con 0x7f58800b4360 2024-03-25T18:45:24.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.524+0000 7f587d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:24.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.524+0000 7f587d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:45:24.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.524+0000 7f587d7fa640 10 monclient(hunting): dump: 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:24.526 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.524+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 141988631 0 0) 0x7f5864004eb0 con 0x7f58800b10d0 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.524+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5880133de0 con 0x7f58800b10d0 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.525+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f586c001970 con 0x7f58800b10d0 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.525+0000 7f587d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.525+0000 7f587d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.525+0000 7f587d7fa640 10 monclient(hunting): dump: 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:24.528 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3756092220 0 0) 0x7f588013cd00 con 0x7f58800a9470 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 >> v1:172.21.15.32:6790/0 conn(0x7f58800b4360 legacy=0x7f58801e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 >> v1:172.21.15.136:6789/0 conn(0x7f58800b10d0 legacy=0x7f588013d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 1 monclient: found mon.a 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f588013cfe0 con 0x7f58800a9470 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:45:24.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f587d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 5 monclient: authenticate success, global_id 4509 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 10 monclient: _renew_subs 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588013bd50 con 0x7f58800a9470 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 10 client.4509.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 10 monclient: _renew_subs 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58801326a0 con 0x7f58800a9470 2024-03-25T18:45:24.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.526+0000 7f5886dd0640 1 librados: init done 2024-03-25T18:45:24.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.527+0000 7f5886dd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:24.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.527+0000 7f5886dd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:24.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.527+0000 7f5886dd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:24.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.527+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58801329a0 con 0x7f58800a9470 2024-03-25T18:45:24.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.840+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5874001ca0 con 0x7f58800a9470 2024-03-25T18:45:24.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.840+0000 7f587d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:24.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.840+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5874003a40 con 0x7f58800a9470 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.840+0000 7f587d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.840+0000 7f587d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.840+0000 7f587d7fa640 10 monclient: dump: 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:24.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.840+0000 7f587effd640 4 set_mon_vals no callback set 2024-03-25T18:45:24.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.841+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5874019a10 con 0x7f58800a9470 2024-03-25T18:45:24.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.842+0000 7f587ffff640 10 monclient: get_auth_request con 0x7f586406e060 auth_method 0 2024-03-25T18:45:24.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.842+0000 7f587d7fa640 10 client.4509.objecter ms_handle_connect 0x7f586406e060 2024-03-25T18:45:24.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.849+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f58740535e0 con 0x7f58800a9470 2024-03-25T18:45:24.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.849+0000 7f587d7fa640 10 client.4509.objecter ms_dispatch 0x7f5880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:45:24.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.849+0000 7f587d7fa640 3 client.4509.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:45:24.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.849+0000 7f587d7fa640 3 client.4509.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:45:24.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.849+0000 7f587d7fa640 20 client.4509.objecter dump_active .. 0 homeless 2024-03-25T18:45:24.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.849+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f587404ef20 con 0x7f58800a9470 2024-03-25T18:45:24.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.849+0000 7f587d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:24.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:24.849+0000 7f587d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:45:25.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.106+0000 7f5886dd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:25.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.106+0000 7f5886dd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:25.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.106+0000 7f5886dd0640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:25.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.106+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58800627b0 con 0x7f58800a9470 2024-03-25T18:45:25.131 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.130+0000 7f587d7fa640 1 -- 172.21.15.32:0/1217085402 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5874001210 con 0x7f58800a9470 2024-03-25T18:45:25.131 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.130+0000 7f587d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:25.131 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.130+0000 7f587d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:45:25.131 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:45:25.132 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:45:25.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.133+0000 7f5886dd0640 10 librados: watch_flush enter 2024-03-25T18:45:25.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.133+0000 7f5886dd0640 10 librados: watch_flush exit 2024-03-25T18:45:25.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.133+0000 7f5886dd0640 20 client.4509.objecter shutdown clearing up homeless session... 2024-03-25T18:45:25.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.133+0000 7f5886dd0640 10 client.4509.objecter successfully canceled tick 2024-03-25T18:45:25.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.133+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 >> v1:172.21.15.136:6800/1141502808 conn(0x7f586406e060 legacy=0x7f5864070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:25.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.133+0000 7f5886dd0640 10 monclient: shutdown 2024-03-25T18:45:25.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.133+0000 7f5886dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:25.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.133+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 >> v1:172.21.15.32:6789/0 conn(0x7f58800a9470 legacy=0x7f588013b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:25.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.134+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 shutdown_connections 2024-03-25T18:45:25.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.134+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 >> 172.21.15.32:0/1217085402 conn(0x7f58800617a0 msgr2=0x7f5880074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:25.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.134+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 shutdown_connections 2024-03-25T18:45:25.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.134+0000 7f5886dd0640 1 -- 172.21.15.32:0/1217085402 wait complete. 2024-03-25T18:45:25.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:25.134+0000 7f5886dd0640 1 librados: shutdown 2024-03-25T18:45:25.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:45:28.083 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:40.410 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:28.093 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:40.420 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:28.128 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:06:40.456 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:28.137 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:40.454 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:28.145 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:40.473 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:28.150 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:40.467 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:28.159 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:40.487 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:28.191 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:06:40.518 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:28.195 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:06:40.512 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:31.953 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:45:32.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.069+0000 7fbc226f1640 10 monclient: get_monmap_and_config 2024-03-25T18:45:32.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.069+0000 7fbc226f1640 10 monclient: build_initial_monmap 2024-03-25T18:45:32.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.069+0000 7fbc226f1640 10 monclient: monmap: 2024-03-25T18:45:32.070 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:32.070 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:32.070 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:45:32.070410+0000 2024-03-25T18:45:32.073 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:45:32.070410+0000 2024-03-25T18:45:32.074 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:32.074 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:32.074 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:32.074 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:45:32.074 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:45:32.074 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.070+0000 7fbc226f1640 1 Processor -- start 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.070+0000 7fbc226f1640 1 -- start start 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.070+0000 7fbc226f1640 10 monclient: init 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 10 monclient(hunting): picked mon.noname-b con 0x7fbc1c0b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 10 monclient(hunting): picked mon.noname-c con 0x7fbc1c0b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1c14be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:32.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1c0af940 con 0x7fbc1c14be90 2024-03-25T18:45:32.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1c0b0a70 con 0x7fbc1c0b1730 2024-03-25T18:45:32.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1c0b0d50 con 0x7fbc1c0b4d60 2024-03-25T18:45:32.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:32.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.071+0000 7fbc226f1640 10 monclient(hunting): authenticate will time out at 6438.260289s 2024-03-25T18:45:32.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.079+0000 7fbc1b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbc1c0b1730 0x7fbc1c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43930/0 (socket says 172.21.15.32:43930) 2024-03-25T18:45:32.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.079+0000 7fbc1b7fe640 1 -- 172.21.15.32:0/2433220639 learned_addr learned my addr 172.21.15.32:0/2433220639 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:32.094 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.094+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2146651311 0 0) 0x7fbc1c0b0a70 con 0x7fbc1c0b1730 2024-03-25T18:45:32.094 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.094+0000 7fbc1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:45:32.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.094+0000 7fbc1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:32.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.094+0000 7fbc1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:32.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.094+0000 7fbc1affd640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:45:32.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.094+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc04003520 con 0x7fbc1c0b1730 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 413483545 0 0) 0x7fbc04003520 con 0x7fbc1c0b1730 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 >> v1:172.21.15.32:6790/0 conn(0x7fbc1c0b4d60 legacy=0x7fbc1c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 >> v1:172.21.15.32:6789/0 conn(0x7fbc1c14be90 legacy=0x7fbc1c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 1 monclient: found mon.noname-c 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c0b1030 con 0x7fbc1c0b1730 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 10 monclient: _finish_auth 0 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:45:32.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbc10001690 con 0x7fbc1c0b1730 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc1affd640 10 monclient: dump: 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:32.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc226f1640 5 monclient: authenticate success, global_id 4495 2024-03-25T18:45:32.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.106+0000 7fbc226f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:32.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.128+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc100017f0 con 0x7fbc1c0b1730 2024-03-25T18:45:32.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.128+0000 7fbc1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:32.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.128+0000 7fbc1affd640 1 -- 172.21.15.32:0/2433220639 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbc10001c00 con 0x7fbc1c0b1730 2024-03-25T18:45:32.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.128+0000 7fbc226f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.128+0000 7fbc226f1640 4 set_mon_vals no callback set 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.128+0000 7fbc1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.128+0000 7fbc1affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.128+0000 7fbc1affd640 10 monclient: dump: 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:32.129 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:32.132 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.129+0000 7fbc226f1640 10 monclient: shutdown 2024-03-25T18:45:32.132 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.129+0000 7fbc226f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.129+0000 7fbc226f1640 1 -- 172.21.15.32:0/2433220639 >> v1:172.21.15.136:6789/0 conn(0x7fbc1c0b1730 legacy=0x7fbc1c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.129+0000 7fbc226f1640 1 -- 172.21.15.32:0/2433220639 shutdown_connections 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.129+0000 7fbc226f1640 1 -- 172.21.15.32:0/2433220639 >> 172.21.15.32:0/2433220639 conn(0x7fbc1c0617a0 msgr2=0x7fbc1c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.129+0000 7fbc226f1640 1 -- 172.21.15.32:0/2433220639 shutdown_connections 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.129+0000 7fbc226f1640 1 -- 172.21.15.32:0/2433220639 wait complete. 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 10 monclient: build_initial_monmap 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 10 monclient: monmap: 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:45:32.133 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 1 librados: starting msgr at 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 1 librados: starting objecter 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 1 Processor -- start 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 1 -- start start 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 1 librados: setting wanted keys 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 1 librados: calling monclient init 2024-03-25T18:45:32.134 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.130+0000 7fbc226f1640 10 monclient: init 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 10 monclient(hunting): picked mon.noname-b con 0x7fbc1c0b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 10 monclient(hunting): picked mon.noname-a con 0x7fbc1c0b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 10 monclient(hunting): picked mon.noname-c con 0x7fbc1c14be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1c135810 con 0x7fbc1c0b4d60 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1c144c10 con 0x7fbc1c0b1730 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc1c144ef0 con 0x7fbc1c14be90 2024-03-25T18:45:32.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:32.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.131+0000 7fbc226f1640 10 monclient(hunting): authenticate will time out at 6438.320217s 2024-03-25T18:45:32.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.139+0000 7fbc20c8e640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fbc1c14be90 0x7fbc1c135100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:33924/0 (socket says 172.21.15.32:33924) 2024-03-25T18:45:32.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.139+0000 7fbc20c8e640 1 -- 172.21.15.32:0/745491253 learned_addr learned my addr 172.21.15.32:0/745491253 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:32.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.144+0000 7fbc18ff9640 10 client.?.objecter ms_handle_connect 0x7fbc1c0b1730 2024-03-25T18:45:32.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.144+0000 7fbc18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:32.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.146+0000 7fbc18ff9640 10 client.?.objecter ms_handle_connect 0x7fbc1c14be90 2024-03-25T18:45:32.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.146+0000 7fbc18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:32.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.151+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941055493 0 0) 0x7fbc1c144ef0 con 0x7fbc1c14be90 2024-03-25T18:45:32.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.151+0000 7fbc18ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:45:32.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.151+0000 7fbc18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:32.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.151+0000 7fbc18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:32.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.151+0000 7fbc18ff9640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:45:32.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.151+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc08004eb0 con 0x7fbc1c14be90 2024-03-25T18:45:32.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.153+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748158998 0 0) 0x7fbc1c144c10 con 0x7fbc1c0b1730 2024-03-25T18:45:32.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.153+0000 7fbc18ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:45:32.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.153+0000 7fbc18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:32.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.153+0000 7fbc18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:32.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.153+0000 7fbc18ff9640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:45:32.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.153+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc1c144ef0 con 0x7fbc1c0b1730 2024-03-25T18:45:32.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.154+0000 7fbc18ff9640 10 client.?.objecter ms_handle_connect 0x7fbc1c0b4d60 2024-03-25T18:45:32.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.154+0000 7fbc18ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:32.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.168+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2324857940 0 0) 0x7fbc08004eb0 con 0x7fbc1c14be90 2024-03-25T18:45:32.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.169+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc1c144c10 con 0x7fbc1c14be90 2024-03-25T18:45:32.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.169+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbc0c001910 con 0x7fbc1c14be90 2024-03-25T18:45:32.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.169+0000 7fbc18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:32.171 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.169+0000 7fbc18ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:45:32.171 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.169+0000 7fbc18ff9640 10 monclient(hunting): dump: 2024-03-25T18:45:32.171 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:32.171 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:32.171 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.171 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.171 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:32.172 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:32.172 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:32.172 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:32.172 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:32.172 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:32.172 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.169+0000 7fbc18ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:45:32.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.174+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552398078 0 0) 0x7fbc1c135810 con 0x7fbc1c0b4d60 2024-03-25T18:45:32.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.174+0000 7fbc18ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:45:32.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.174+0000 7fbc18ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:32.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.174+0000 7fbc18ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:32.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.174+0000 7fbc18ff9640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:45:32.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.174+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc08004eb0 con 0x7fbc1c0b4d60 2024-03-25T18:45:32.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.175+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1788402133 0 0) 0x7fbc1c144ef0 con 0x7fbc1c0b1730 2024-03-25T18:45:32.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.175+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc1c135810 con 0x7fbc1c0b1730 2024-03-25T18:45:32.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.175+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbc14001c20 con 0x7fbc1c0b1730 2024-03-25T18:45:32.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.175+0000 7fbc18ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:32.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.175+0000 7fbc18ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:45:32.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.175+0000 7fbc18ff9640 10 monclient(hunting): dump: 2024-03-25T18:45:32.177 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:32.177 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:32.177 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.178 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.178 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:32.178 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:32.178 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:32.178 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:32.178 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:32.179 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:32.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3162420251 0 0) 0x7fbc1c144c10 con 0x7fbc1c14be90 2024-03-25T18:45:32.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 >> v1:172.21.15.136:6789/0 conn(0x7fbc1c0b1730 legacy=0x7fbc1c1d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:32.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 >> v1:172.21.15.32:6789/0 conn(0x7fbc1c0b4d60 legacy=0x7fbc1c1349f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:32.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:45:32.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 1 monclient: found mon.c 2024-03-25T18:45:32.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:32.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:45:32.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c140c10 con 0x7fbc1c14be90 2024-03-25T18:45:32.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 10 monclient: _finish_auth 0 2024-03-25T18:45:32.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:45:32.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc18ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:32.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 5 monclient: authenticate success, global_id 4502 2024-03-25T18:45:32.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 10 monclient: _renew_subs 2024-03-25T18:45:32.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:45:32.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1c137d50 con 0x7fbc1c14be90 2024-03-25T18:45:32.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:45:32.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 10 monclient: _renew_subs 2024-03-25T18:45:32.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:45:32.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1c1407d0 con 0x7fbc1c14be90 2024-03-25T18:45:32.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.186+0000 7fbc226f1640 1 librados: init done 2024-03-25T18:45:32.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.187+0000 7fbc226f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:32.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.187+0000 7fbc226f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:32.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.187+0000 7fbc226f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:45:32.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.187+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc1c141e30 con 0x7fbc1c14be90 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.515+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc0c003610 con 0x7fbc1c14be90 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.515+0000 7fbc18ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.515+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbc0c0039e0 con 0x7fbc1c14be90 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.515+0000 7fbc18ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.515+0000 7fbc18ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.515+0000 7fbc18ff9640 10 monclient: dump: 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.516 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:32.517 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:32.517 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:32.517 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:32.517 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:32.517 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:32.517 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:32.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.515+0000 7fbc1a7fc640 4 set_mon_vals no callback set 2024-03-25T18:45:32.518 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.516+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fbc0c003c00 con 0x7fbc1c14be90 2024-03-25T18:45:32.518 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.517+0000 7fbc1bfff640 10 monclient: get_auth_request con 0x7fbc0806dfb0 auth_method 0 2024-03-25T18:45:32.518 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.517+0000 7fbc18ff9640 10 client.4502.objecter ms_handle_connect 0x7fbc0806dfb0 2024-03-25T18:45:32.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.523+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fbc0c0546c0 con 0x7fbc1c14be90 2024-03-25T18:45:32.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.523+0000 7fbc18ff9640 10 client.4502.objecter ms_dispatch 0x7fbc1c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:45:32.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.523+0000 7fbc18ff9640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:45:32.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.523+0000 7fbc18ff9640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:45:32.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.523+0000 7fbc18ff9640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-25T18:45:32.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.523+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbc0c0501d0 con 0x7fbc1c14be90 2024-03-25T18:45:32.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.523+0000 7fbc18ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:32.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.523+0000 7fbc18ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:45:32.752 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.752+0000 7fbc226f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:32.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.752+0000 7fbc226f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:32.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.752+0000 7fbc226f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:45:32.753 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.752+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc1c145080 con 0x7fbc1c14be90 2024-03-25T18:45:32.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.823+0000 7fbc18ff9640 1 -- 172.21.15.32:0/745491253 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fbc0c052100 con 0x7fbc1c14be90 2024-03-25T18:45:32.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.823+0000 7fbc18ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:32.825 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.823+0000 7fbc18ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:45:32.825 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:45:32.825 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:45:32.827 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 10 librados: watch_flush enter 2024-03-25T18:45:32.827 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 10 librados: watch_flush exit 2024-03-25T18:45:32.827 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-25T18:45:32.827 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 10 client.4502.objecter successfully canceled tick 2024-03-25T18:45:32.827 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 >> v1:172.21.15.136:6800/1141502808 conn(0x7fbc0806dfb0 legacy=0x7fbc08070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:32.827 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 10 monclient: shutdown 2024-03-25T18:45:32.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:32.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 >> v1:172.21.15.32:6790/0 conn(0x7fbc1c14be90 legacy=0x7fbc1c135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:32.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 shutdown_connections 2024-03-25T18:45:32.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 >> 172.21.15.32:0/745491253 conn(0x7fbc1c0617a0 msgr2=0x7fbc1c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:32.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 shutdown_connections 2024-03-25T18:45:32.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 1 -- 172.21.15.32:0/745491253 wait complete. 2024-03-25T18:45:32.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:32.826+0000 7fbc226f1640 1 librados: shutdown 2024-03-25T18:45:32.839 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:45:39.647 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:45:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.761+0000 7f5a2af90640 10 monclient: get_monmap_and_config 2024-03-25T18:45:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.761+0000 7f5a2af90640 10 monclient: build_initial_monmap 2024-03-25T18:45:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.761+0000 7f5a2af90640 10 monclient: monmap: 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:45:39.761721+0000 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:45:39.761721+0000 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:45:39.762 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.762+0000 7f5a2af90640 1 Processor -- start 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.762+0000 7f5a2af90640 1 -- start start 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.762+0000 7f5a2af90640 10 monclient: init 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.762+0000 7f5a2af90640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.762+0000 7f5a2af90640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.762+0000 7f5a2af90640 10 monclient(hunting): picked mon.noname-b con 0x7f5a240a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.762+0000 7f5a2af90640 10 monclient(hunting): picked mon.noname-a con 0x7f5a2413f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.762+0000 7f5a2af90640 10 monclient(hunting): picked mon.noname-c con 0x7f5a241430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.763+0000 7f5a2af90640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a24142520 con 0x7f5a2413f710 2024-03-25T18:45:39.763 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.763+0000 7f5a2af90640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a240b28a0 con 0x7f5a241430c0 2024-03-25T18:45:39.764 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.763+0000 7f5a2af90640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a240b2a30 con 0x7f5a240a98f0 2024-03-25T18:45:39.764 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.763+0000 7f5a2af90640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:39.764 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.763+0000 7f5a2af90640 10 monclient(hunting): authenticate will time out at 6445.951559s 2024-03-25T18:45:39.771 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.771+0000 7f5a28d2c640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f5a240a98f0 0x7f5a240a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:57444/0 (socket says 172.21.15.32:57444) 2024-03-25T18:45:39.771 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.771+0000 7f5a28d2c640 1 -- 172.21.15.32:0/32245450 learned_addr learned my addr 172.21.15.32:0/32245450 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:39.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.782+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2468035635 0 0) 0x7f5a24142520 con 0x7f5a2413f710 2024-03-25T18:45:39.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.782+0000 7f5a237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:39.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.782+0000 7f5a237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:39.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.782+0000 7f5a237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:39.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.782+0000 7f5a237fe640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:45:39.782 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.782+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a14003520 con 0x7f5a2413f710 2024-03-25T18:45:39.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.788+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1162588497 0 0) 0x7f5a240b2a30 con 0x7f5a240a98f0 2024-03-25T18:45:39.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.788+0000 7f5a237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:39.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.788+0000 7f5a237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:39.788 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.788+0000 7f5a237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:39.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.788+0000 7f5a237fe640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:45:39.789 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.788+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a24142520 con 0x7f5a240a98f0 2024-03-25T18:45:39.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.795+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3342244862 0 0) 0x7f5a240b28a0 con 0x7f5a241430c0 2024-03-25T18:45:39.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.795+0000 7f5a237fe640 10 monclient(hunting): _negotiate 2024-03-25T18:45:39.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.795+0000 7f5a237fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:39.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.795+0000 7f5a237fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:39.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.795+0000 7f5a237fe640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:45:39.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.795+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a240b2a30 con 0x7f5a241430c0 2024-03-25T18:45:39.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2296722279 0 0) 0x7f5a14003520 con 0x7f5a2413f710 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 >> v1:172.21.15.32:6790/0 conn(0x7f5a240a98f0 legacy=0x7f5a240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 >> v1:172.21.15.136:6789/0 conn(0x7f5a241430c0 legacy=0x7f5a2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 10 monclient: _finish_hunting 0 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 1 monclient: found mon.noname-a 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a240b44c0 con 0x7f5a2413f710 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 10 monclient: _finish_auth 0 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 10 monclient: _check_auth_tickets 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5a10001690 con 0x7f5a2413f710 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a2af90640 5 monclient: authenticate success, global_id 4518 2024-03-25T18:45:39.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a2af90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a237fe640 10 monclient: dump: 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:39.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.805+0000 7f5a2af90640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:39.816 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.816+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a100017f0 con 0x7f5a2413f710 2024-03-25T18:45:39.816 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.816+0000 7f5a237fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.816+0000 7f5a237fe640 1 -- 172.21.15.32:0/32245450 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5a10001c00 con 0x7f5a2413f710 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.816+0000 7f5a2af90640 10 monclient: get_monmap_and_config success 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.816+0000 7f5a2af90640 4 set_mon_vals no callback set 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.816+0000 7f5a237fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.816+0000 7f5a237fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.816+0000 7f5a237fe640 10 monclient: dump: 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:39.817 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:39.818 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:39.818 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.817+0000 7f5a2af90640 10 monclient: shutdown 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.817+0000 7f5a2af90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.817+0000 7f5a2af90640 1 -- 172.21.15.32:0/32245450 >> v1:172.21.15.32:6789/0 conn(0x7f5a2413f710 legacy=0x7f5a241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.817+0000 7f5a2af90640 1 -- 172.21.15.32:0/32245450 shutdown_connections 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.817+0000 7f5a2af90640 1 -- 172.21.15.32:0/32245450 >> 172.21.15.32:0/32245450 conn(0x7f5a240b7ea0 msgr2=0x7f5a240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.817+0000 7f5a2af90640 1 -- 172.21.15.32:0/32245450 shutdown_connections 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.817+0000 7f5a2af90640 1 -- 172.21.15.32:0/32245450 wait complete. 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.818+0000 7f5a2af90640 10 monclient: build_initial_monmap 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.818+0000 7f5a2af90640 10 monclient: monmap: 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:39.820 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.818+0000 7f5a2af90640 1 librados: starting msgr at 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.818+0000 7f5a2af90640 1 librados: starting objecter 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.818+0000 7f5a2af90640 1 Processor -- start 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 1 -- start start 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 1 librados: setting wanted keys 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 1 librados: calling monclient init 2024-03-25T18:45:39.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 10 monclient: init 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 10 monclient(hunting): picked mon.noname-c con 0x7f5a240a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 10 monclient(hunting): picked mon.noname-a con 0x7f5a2413f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 10 monclient(hunting): picked mon.noname-b con 0x7f5a241430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a24139520 con 0x7f5a2413f710 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a24138b40 con 0x7f5a241430c0 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a24138e20 con 0x7f5a240a98f0 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:39.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.819+0000 7f5a2af90640 10 monclient(hunting): authenticate will time out at 6446.008291s 2024-03-25T18:45:39.838 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.838+0000 7f5a23fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5a2413f710 0x7f5a24137b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44658/0 (socket says 172.21.15.32:44658) 2024-03-25T18:45:39.838 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.838+0000 7f5a23fff640 1 -- 172.21.15.32:0/2320400351 learned_addr learned my addr 172.21.15.32:0/2320400351 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:39.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.840+0000 7f5a217fa640 10 client.?.objecter ms_handle_connect 0x7f5a2413f710 2024-03-25T18:45:39.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.840+0000 7f5a217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:39.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.841+0000 7f5a217fa640 10 client.?.objecter ms_handle_connect 0x7f5a241430c0 2024-03-25T18:45:39.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.841+0000 7f5a217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:39.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.846+0000 7f5a217fa640 10 client.?.objecter ms_handle_connect 0x7f5a240a98f0 2024-03-25T18:45:39.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.846+0000 7f5a217fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:39.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.848+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 120776679 0 0) 0x7f5a24139520 con 0x7f5a2413f710 2024-03-25T18:45:39.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.848+0000 7f5a217fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:39.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.848+0000 7f5a217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:39.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.848+0000 7f5a217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:39.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.848+0000 7f5a217fa640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:45:39.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.848+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a08004eb0 con 0x7f5a2413f710 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.854+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2919740440 0 0) 0x7f5a08004eb0 con 0x7f5a2413f710 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a24139520 con 0x7f5a2413f710 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5a10001bc0 con 0x7f5a2413f710 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 10 monclient(hunting): dump: 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:39.855 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2293754311 0 0) 0x7f5a24138b40 con 0x7f5a241430c0 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:45:39.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.855+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a08004eb0 con 0x7f5a241430c0 2024-03-25T18:45:39.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.861+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939726207 0 0) 0x7f5a24138e20 con 0x7f5a240a98f0 2024-03-25T18:45:39.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.861+0000 7f5a217fa640 10 monclient(hunting): _negotiate 2024-03-25T18:45:39.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.861+0000 7f5a217fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:39.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.861+0000 7f5a217fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:39.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.861+0000 7f5a217fa640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:45:39.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.861+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a24138b40 con 0x7f5a240a98f0 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3030299891 0 0) 0x7f5a24139520 con 0x7f5a2413f710 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 >> v1:172.21.15.32:6790/0 conn(0x7f5a240a98f0 legacy=0x7f5a24137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 >> v1:172.21.15.136:6789/0 conn(0x7f5a241430c0 legacy=0x7f5a241e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 10 monclient: _finish_hunting 0 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 1 monclient: found mon.a 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a24139100 con 0x7f5a2413f710 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 10 monclient: _finish_auth 0 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 10 monclient: _check_auth_tickets 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a217fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:39.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 5 monclient: authenticate success, global_id 4521 2024-03-25T18:45:39.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 10 monclient: _renew_subs 2024-03-25T18:45:39.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:39.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a24136aa0 con 0x7f5a2413f710 2024-03-25T18:45:39.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 10 client.4521.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:45:39.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 10 monclient: _renew_subs 2024-03-25T18:45:39.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:39.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a2413ba10 con 0x7f5a2413f710 2024-03-25T18:45:39.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.865+0000 7f5a2af90640 1 librados: init done 2024-03-25T18:45:39.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.866+0000 7f5a2af90640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:39.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.866+0000 7f5a2af90640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:39.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.866+0000 7f5a2af90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:39.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:39.866+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a240627b0 con 0x7f5a2413f710 2024-03-25T18:45:40.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.197+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a100011f0 con 0x7f5a2413f710 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.197+0000 7f5a217fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.197+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5a10003a40 con 0x7f5a2413f710 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.197+0000 7f5a217fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.197+0000 7f5a217fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.197+0000 7f5a217fa640 10 monclient: dump: 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:40.198 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:40.199 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:40.199 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:40.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.197+0000 7f5a22ffd640 4 set_mon_vals no callback set 2024-03-25T18:45:40.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.198+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5a10003c60 con 0x7f5a2413f710 2024-03-25T18:45:40.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.199+0000 7f5a28d2c640 10 monclient: get_auth_request con 0x7f5a0806de10 auth_method 0 2024-03-25T18:45:40.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.199+0000 7f5a217fa640 10 client.4521.objecter ms_handle_connect 0x7f5a0806de10 2024-03-25T18:45:40.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.203+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5a10054920 con 0x7f5a2413f710 2024-03-25T18:45:40.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.203+0000 7f5a217fa640 10 client.4521.objecter ms_dispatch 0x7f5a24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:45:40.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.203+0000 7f5a217fa640 3 client.4521.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:45:40.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.203+0000 7f5a217fa640 3 client.4521.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:45:40.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.204+0000 7f5a217fa640 20 client.4521.objecter dump_active .. 0 homeless 2024-03-25T18:45:40.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.204+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a10050260 con 0x7f5a2413f710 2024-03-25T18:45:40.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.204+0000 7f5a217fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:40.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.204+0000 7f5a217fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:45:40.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.423+0000 7f5a2af90640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:40.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.423+0000 7f5a2af90640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:40.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.423+0000 7f5a2af90640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:45:40.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.423+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a241398c0 con 0x7f5a2413f710 2024-03-25T18:45:40.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.467+0000 7f5a217fa640 1 -- 172.21.15.32:0/2320400351 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5a10052360 con 0x7f5a2413f710 2024-03-25T18:45:40.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.467+0000 7f5a217fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:40.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.467+0000 7f5a217fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:45:40.471 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:45:40.471 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.469+0000 7f5a2af90640 10 librados: watch_flush enter 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 10 librados: watch_flush exit 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 20 client.4521.objecter shutdown clearing up homeless session... 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 10 client.4521.objecter successfully canceled tick 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 >> v1:172.21.15.136:6800/1141502808 conn(0x7f5a0806de10 legacy=0x7f5a080702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 10 monclient: shutdown 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 >> v1:172.21.15.32:6789/0 conn(0x7f5a2413f710 legacy=0x7f5a24137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 shutdown_connections 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 >> 172.21.15.32:0/2320400351 conn(0x7f5a240b7ea0 msgr2=0x7f5a24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:40.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 shutdown_connections 2024-03-25T18:45:40.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 1 -- 172.21.15.32:0/2320400351 wait complete. 2024-03-25T18:45:40.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:40.470+0000 7f5a2af90640 1 librados: shutdown 2024-03-25T18:45:40.481 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:45:47.288 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.407+0000 7f9b80782640 10 monclient: get_monmap_and_config 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.407+0000 7f9b80782640 10 monclient: build_initial_monmap 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.407+0000 7f9b80782640 10 monclient: monmap: 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:45:47.409312+0000 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:45:47.409312+0000 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:47.409 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:47.410 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:47.410 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:45:47.410 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:45:47.410 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.408+0000 7f9b80782640 1 Processor -- start 2024-03-25T18:45:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.408+0000 7f9b80782640 1 -- start start 2024-03-25T18:45:47.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.408+0000 7f9b80782640 10 monclient: init 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 10 monclient(hunting): picked mon.noname-c con 0x7f9b780b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 10 monclient(hunting): picked mon.noname-a con 0x7f9b780b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7814be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780af940 con 0x7f9b780b1730 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780b0a70 con 0x7f9b780b4d60 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b780b0d50 con 0x7f9b7814be90 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:47.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.409+0000 7f9b80782640 10 monclient(hunting): authenticate will time out at 6453.599038s 2024-03-25T18:45:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.420+0000 7f9b7e51e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9b780b4d60 0x7f9b7813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:57338/0 (socket says 172.21.15.32:57338) 2024-03-25T18:45:47.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.420+0000 7f9b7e51e640 1 -- 172.21.15.32:0/126490703 learned_addr learned my addr 172.21.15.32:0/126490703 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.431+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4091335571 0 0) 0x7f9b780b0a70 con 0x7f9b780b4d60 2024-03-25T18:45:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.431+0000 7f9b7d51c640 10 monclient(hunting): _negotiate 2024-03-25T18:45:47.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.431+0000 7f9b7d51c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.431+0000 7f9b7d51c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.431+0000 7f9b7d51c640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:45:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.431+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b64003520 con 0x7f9b780b4d60 2024-03-25T18:45:47.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.432+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610727449 0 0) 0x7f9b780b0d50 con 0x7f9b7814be90 2024-03-25T18:45:47.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.432+0000 7f9b7d51c640 10 monclient(hunting): _negotiate 2024-03-25T18:45:47.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.432+0000 7f9b7d51c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:47.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.432+0000 7f9b7d51c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:47.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.432+0000 7f9b7d51c640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:45:47.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.432+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b780b0a70 con 0x7f9b7814be90 2024-03-25T18:45:47.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.440+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2382070862 0 0) 0x7f9b780af940 con 0x7f9b780b1730 2024-03-25T18:45:47.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.440+0000 7f9b7d51c640 10 monclient(hunting): _negotiate 2024-03-25T18:45:47.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.440+0000 7f9b7d51c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:47.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.440+0000 7f9b7d51c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:47.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.440+0000 7f9b7d51c640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:45:47.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.440+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b780b0d50 con 0x7f9b780b1730 2024-03-25T18:45:47.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750956904 0 0) 0x7f9b780b0a70 con 0x7f9b7814be90 2024-03-25T18:45:47.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 >> v1:172.21.15.136:6789/0 conn(0x7f9b780b4d60 legacy=0x7f9b7813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:47.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 >> v1:172.21.15.32:6789/0 conn(0x7f9b780b1730 legacy=0x7f9b780b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 10 monclient: _finish_hunting 0 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 1 monclient: found mon.noname-b 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b780b1030 con 0x7f9b7814be90 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 10 monclient: _finish_auth 0 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 10 monclient: _check_auth_tickets 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9b700019f0 con 0x7f9b7814be90 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b7d51c640 10 monclient: dump: 2024-03-25T18:45:47.455 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b80782640 5 monclient: authenticate success, global_id 4511 2024-03-25T18:45:47.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.453+0000 7f9b80782640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:47.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.468+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b70001b50 con 0x7f9b7814be90 2024-03-25T18:45:47.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.468+0000 7f9b7d51c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.468+0000 7f9b7d51c640 1 -- 172.21.15.32:0/126490703 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9b70001f60 con 0x7f9b7814be90 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.468+0000 7f9b80782640 10 monclient: get_monmap_and_config success 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.468+0000 7f9b80782640 4 set_mon_vals no callback set 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.468+0000 7f9b7d51c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.468+0000 7f9b7d51c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.468+0000 7f9b7d51c640 10 monclient: dump: 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:47.470 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:47.471 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:47.471 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:47.471 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:47.471 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.469+0000 7f9b80782640 10 monclient: shutdown 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.469+0000 7f9b80782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.469+0000 7f9b80782640 1 -- 172.21.15.32:0/126490703 >> v1:172.21.15.32:6790/0 conn(0x7f9b7814be90 legacy=0x7f9b7814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.469+0000 7f9b80782640 1 -- 172.21.15.32:0/126490703 shutdown_connections 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.469+0000 7f9b80782640 1 -- 172.21.15.32:0/126490703 >> 172.21.15.32:0/126490703 conn(0x7f9b780617a0 msgr2=0x7f9b78061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.469+0000 7f9b80782640 1 -- 172.21.15.32:0/126490703 shutdown_connections 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.469+0000 7f9b80782640 1 -- 172.21.15.32:0/126490703 wait complete. 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.470+0000 7f9b80782640 10 monclient: build_initial_monmap 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.470+0000 7f9b80782640 10 monclient: monmap: 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:45:47.473 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.470+0000 7f9b80782640 1 librados: starting msgr at 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.470+0000 7f9b80782640 1 librados: starting objecter 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 1 Processor -- start 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 1 -- start start 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 1 librados: setting wanted keys 2024-03-25T18:45:47.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 1 librados: calling monclient init 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 10 monclient: init 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 10 monclient(hunting): picked mon.noname-a con 0x7f9b780b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 10 monclient(hunting): picked mon.noname-c con 0x7f9b780b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 10 monclient(hunting): picked mon.noname-b con 0x7f9b7814be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b781419c0 con 0x7f9b780b1730 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7813bbc0 con 0x7f9b7814be90 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9b7813bd50 con 0x7f9b780b4d60 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:47.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.471+0000 7f9b80782640 10 monclient(hunting): authenticate will time out at 6453.661210s 2024-03-25T18:45:47.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.482+0000 7f9b7ed1f640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f9b7814be90 0x7f9b781e20d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:57384/0 (socket says 172.21.15.32:57384) 2024-03-25T18:45:47.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.482+0000 7f9b7ed1f640 1 -- 172.21.15.32:0/2011959962 learned_addr learned my addr 172.21.15.32:0/2011959962 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:47.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.485+0000 7f9b6effd640 10 client.?.objecter ms_handle_connect 0x7f9b7814be90 2024-03-25T18:45:47.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.485+0000 7f9b6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:47.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.488+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2341819048 0 0) 0x7f9b7813bbc0 con 0x7f9b7814be90 2024-03-25T18:45:47.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.488+0000 7f9b6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:45:47.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.488+0000 7f9b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:47.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.488+0000 7f9b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:47.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.488+0000 7f9b6effd640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:45:47.489 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.488+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c004eb0 con 0x7f9b7814be90 2024-03-25T18:45:47.491 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.490+0000 7f9b6effd640 10 client.?.objecter ms_handle_connect 0x7f9b780b4d60 2024-03-25T18:45:47.491 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.490+0000 7f9b6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:47.492 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.491+0000 7f9b6effd640 10 client.?.objecter ms_handle_connect 0x7f9b780b1730 2024-03-25T18:45:47.492 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.491+0000 7f9b6effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:47.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3514011648 0 0) 0x7f9b5c004eb0 con 0x7f9b7814be90 2024-03-25T18:45:47.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b7813bbc0 con 0x7f9b7814be90 2024-03-25T18:45:47.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9b70001fa0 con 0x7f9b7814be90 2024-03-25T18:45:47.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 10 monclient(hunting): dump: 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:47.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:45:47.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240432806 0 0) 0x7f9b781419c0 con 0x7f9b780b1730 2024-03-25T18:45:47.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:45:47.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:47.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:47.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:45:47.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.509+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b5c004eb0 con 0x7f9b780b1730 2024-03-25T18:45:47.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.515+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 903931662 0 0) 0x7f9b7813bd50 con 0x7f9b780b4d60 2024-03-25T18:45:47.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.515+0000 7f9b6effd640 10 monclient(hunting): _negotiate 2024-03-25T18:45:47.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.515+0000 7f9b6effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:47.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.515+0000 7f9b6effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:47.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.515+0000 7f9b6effd640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:45:47.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.515+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9b781419c0 con 0x7f9b780b4d60 2024-03-25T18:45:47.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.520+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1395620047 0 0) 0x7f9b5c004eb0 con 0x7f9b780b1730 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.520+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9b7813bd50 con 0x7f9b780b1730 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.520+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9b740017b0 con 0x7f9b780b1730 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.520+0000 7f9b6effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.520+0000 7f9b6effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.520+0000 7f9b6effd640 10 monclient(hunting): dump: 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:47.522 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:47.523 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:47.523 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:47.532 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4193858163 0 0) 0x7f9b7813bbc0 con 0x7f9b7814be90 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 >> v1:172.21.15.32:6790/0 conn(0x7f9b780b4d60 legacy=0x7f9b781d7ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 >> v1:172.21.15.32:6789/0 conn(0x7f9b780b1730 legacy=0x7f9b781405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 10 monclient: _finish_hunting 0 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 1 monclient: found mon.b 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b7813b060 con 0x7f9b7814be90 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 10 monclient: _finish_auth 0 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 10 monclient: _check_auth_tickets 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b6effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 5 monclient: authenticate success, global_id 4510 2024-03-25T18:45:47.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 10 monclient: _renew_subs 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b781421a0 con 0x7f9b7814be90 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 10 client.4510.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 10 monclient: _renew_subs 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b781d8590 con 0x7f9b7814be90 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.530+0000 7f9b80782640 1 librados: init done 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.531+0000 7f9b80782640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.531+0000 7f9b80782640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.531+0000 7f9b80782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:47.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.531+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b7813b250 con 0x7f9b7814be90 2024-03-25T18:45:47.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.810+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9b70003c20 con 0x7f9b7814be90 2024-03-25T18:45:47.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.810+0000 7f9b6effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:47.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.810+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f9b70004030 con 0x7f9b7814be90 2024-03-25T18:45:47.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.810+0000 7f9b6effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:47.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.810+0000 7f9b6effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:45:47.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.810+0000 7f9b6effd640 10 monclient: dump: 2024-03-25T18:45:47.812 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:47.812 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:47.813 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.810+0000 7f9b7cd1b640 4 set_mon_vals no callback set 2024-03-25T18:45:47.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.812+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f9b70004210 con 0x7f9b7814be90 2024-03-25T18:45:47.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.812+0000 7f9b7e51e640 10 monclient: get_auth_request con 0x7f9b5c06e020 auth_method 0 2024-03-25T18:45:47.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.813+0000 7f9b6effd640 10 client.4510.objecter ms_handle_connect 0x7f9b5c06e020 2024-03-25T18:45:47.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.819+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f9b70054b80 con 0x7f9b7814be90 2024-03-25T18:45:47.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.819+0000 7f9b6effd640 10 client.4510.objecter ms_dispatch 0x7f9b78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:45:47.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.819+0000 7f9b6effd640 3 client.4510.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:45:47.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.819+0000 7f9b6effd640 3 client.4510.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:45:47.820 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.819+0000 7f9b6effd640 20 client.4510.objecter dump_active .. 0 homeless 2024-03-25T18:45:47.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.820+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9b70050690 con 0x7f9b7814be90 2024-03-25T18:45:47.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.820+0000 7f9b6effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:47.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:47.820+0000 7f9b6effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:45:48.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.075+0000 7f9b80782640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:48.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.075+0000 7f9b80782640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:48.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.075+0000 7f9b80782640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:48.077 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.075+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9b780627b0 con 0x7f9b7814be90 2024-03-25T18:45:48.144 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:00.471 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:48.161 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:00.488 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:48.197 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:00.524 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:48.228 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:00.541 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:48.229 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:00.533 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:48.229 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:00.538 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:48.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.217+0000 7f9b6effd640 1 -- 172.21.15.32:0/2011959962 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f9b700525c0 con 0x7f9b7814be90 2024-03-25T18:45:48.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.217+0000 7f9b6effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:48.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.217+0000 7f9b6effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:45:48.231 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:45:48.231 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:45:48.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 10 librados: watch_flush enter 2024-03-25T18:45:48.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 10 librados: watch_flush exit 2024-03-25T18:45:48.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 20 client.4510.objecter shutdown clearing up homeless session... 2024-03-25T18:45:48.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 10 client.4510.objecter successfully canceled tick 2024-03-25T18:45:48.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 >> v1:172.21.15.136:6800/1141502808 conn(0x7f9b5c06e020 legacy=0x7f9b5c0704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:48.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 10 monclient: shutdown 2024-03-25T18:45:48.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:48.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 >> v1:172.21.15.136:6789/0 conn(0x7f9b7814be90 legacy=0x7f9b781e20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:48.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 shutdown_connections 2024-03-25T18:45:48.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 >> 172.21.15.32:0/2011959962 conn(0x7f9b780617a0 msgr2=0x7f9b7813c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:48.235 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 shutdown_connections 2024-03-25T18:45:48.235 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 1 -- 172.21.15.32:0/2011959962 wait complete. 2024-03-25T18:45:48.235 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:48.219+0000 7f9b80782640 1 librados: shutdown 2024-03-25T18:45:48.235 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:00.557 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:48.237 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:45:48.244 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:00.561 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:48.259 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:00.587 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:45:55.043 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:45:55.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.159+0000 7faa0230e640 10 monclient: get_monmap_and_config 2024-03-25T18:45:55.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.159+0000 7faa0230e640 10 monclient: build_initial_monmap 2024-03-25T18:45:55.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.159+0000 7faa0230e640 10 monclient: monmap: 2024-03-25T18:45:55.160 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:45:55.160689+0000 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:45:55.160689+0000 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:45:55.161 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:55.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 1 Processor -- start 2024-03-25T18:45:55.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 1 -- start start 2024-03-25T18:45:55.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 10 monclient: init 2024-03-25T18:45:55.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:55.162 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:45:55.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9fc0a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:55.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 10 monclient(hunting): picked mon.noname-c con 0x7fa9fc13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:55.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 10 monclient(hunting): picked mon.noname-b con 0x7fa9fc1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:55.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc142520 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0b28a0 con 0x7fa9fc13f710 2024-03-25T18:45:55.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc0b2a30 con 0x7fa9fc1430c0 2024-03-25T18:45:55.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:55.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.160+0000 7faa0230e640 10 monclient(hunting): authenticate will time out at 6461.350462s 2024-03-25T18:45:55.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.169+0000 7fa9fb7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa9fc13f710 0x7fa9fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:33630/0 (socket says 172.21.15.32:33630) 2024-03-25T18:45:55.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.169+0000 7fa9fb7fe640 1 -- 172.21.15.32:0/3501323475 learned_addr learned my addr 172.21.15.32:0/3501323475 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:55.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.180+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 218682801 0 0) 0x7fa9fc0b28a0 con 0x7fa9fc13f710 2024-03-25T18:45:55.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.180+0000 7fa9faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:45:55.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.180+0000 7fa9faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:55.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.180+0000 7fa9faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:55.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.180+0000 7fa9faffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:45:55.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.180+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e4003520 con 0x7fa9fc13f710 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 210311275 0 0) 0x7fa9e4003520 con 0x7fa9fc13f710 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 >> v1:172.21.15.32:6790/0 conn(0x7fa9fc1430c0 legacy=0x7fa9fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 >> v1:172.21.15.32:6789/0 conn(0x7fa9fc0a98f0 legacy=0x7fa9fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 10 monclient: _finish_hunting 0 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 1 monclient: found mon.noname-c 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc0b44c0 con 0x7fa9fc13f710 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 10 monclient: _finish_auth 0 2024-03-25T18:45:55.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 10 monclient: _check_auth_tickets 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa9f0001690 con 0x7fa9fc13f710 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7fa9faffd640 10 monclient: dump: 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:55.199 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:55.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7faa0230e640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:45:55.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.196+0000 7faa0230e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9f00017f0 con 0x7fa9fc13f710 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7fa9faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7fa9faffd640 1 -- 172.21.15.32:0/3501323475 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa9f0001c00 con 0x7fa9fc13f710 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7faa0230e640 10 monclient: get_monmap_and_config success 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7faa0230e640 4 set_mon_vals no callback set 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7fa9faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7fa9faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7fa9faffd640 10 monclient: dump: 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.207 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:55.208 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:55.208 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:55.208 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:55.208 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:55.208 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7faa0230e640 10 monclient: shutdown 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7faa0230e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.206+0000 7faa0230e640 1 -- 172.21.15.32:0/3501323475 >> v1:172.21.15.136:6789/0 conn(0x7fa9fc13f710 legacy=0x7fa9fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.207+0000 7faa0230e640 1 -- 172.21.15.32:0/3501323475 shutdown_connections 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.207+0000 7faa0230e640 1 -- 172.21.15.32:0/3501323475 >> 172.21.15.32:0/3501323475 conn(0x7fa9fc0b7ea0 msgr2=0x7fa9fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.207+0000 7faa0230e640 1 -- 172.21.15.32:0/3501323475 shutdown_connections 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.207+0000 7faa0230e640 1 -- 172.21.15.32:0/3501323475 wait complete. 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.207+0000 7faa0230e640 10 monclient: build_initial_monmap 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.207+0000 7faa0230e640 10 monclient: monmap: 2024-03-25T18:45:55.209 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 1 librados: starting msgr at 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 1 librados: starting objecter 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 1 Processor -- start 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 1 -- start start 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 1 librados: setting wanted keys 2024-03-25T18:45:55.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 1 librados: calling monclient init 2024-03-25T18:45:55.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 10 monclient: init 2024-03-25T18:45:55.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 10 monclient: _reopen_session rank -1 2024-03-25T18:45:55.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.208+0000 7faa0230e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:45:55.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.209+0000 7faa0230e640 10 monclient(hunting): picked mon.noname-b con 0x7fa9fc0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:45:55.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.209+0000 7faa0230e640 10 monclient(hunting): picked mon.noname-a con 0x7fa9fc13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:45:55.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.209+0000 7faa0230e640 10 monclient(hunting): picked mon.noname-c con 0x7fa9fc1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:45:55.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.209+0000 7faa0230e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc1397d0 con 0x7fa9fc13f710 2024-03-25T18:45:55.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.209+0000 7faa0230e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc1d7500 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.209+0000 7faa0230e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9fc134de0 con 0x7fa9fc1430c0 2024-03-25T18:45:55.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.209+0000 7faa0230e640 10 monclient(hunting): _renew_subs 2024-03-25T18:45:55.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.209+0000 7faa0230e640 10 monclient(hunting): authenticate will time out at 6461.398601s 2024-03-25T18:45:55.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.217+0000 7fa9fbfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fa9fc0a98f0 0x7fa9fc138420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:33654/0 (socket says 172.21.15.32:33654) 2024-03-25T18:45:55.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.217+0000 7fa9fbfff640 1 -- 172.21.15.32:0/2892066089 learned_addr learned my addr 172.21.15.32:0/2892066089 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:45:55.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.220+0000 7fa9f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9fc0a98f0 2024-03-25T18:45:55.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.220+0000 7fa9f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:55.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.222+0000 7fa9f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9fc1430c0 2024-03-25T18:45:55.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.222+0000 7fa9f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:55.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.230+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2608301500 0 0) 0x7fa9fc1d7500 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.230+0000 7fa9f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:45:55.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.230+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:55.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.230+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:55.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.230+0000 7fa9f8ff9640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:45:55.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.230+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9e8004eb0 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.233+0000 7fa9f8ff9640 10 client.?.objecter ms_handle_connect 0x7fa9fc13f710 2024-03-25T18:45:55.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.233+0000 7fa9f8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:45:55.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.242+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1011237356 0 0) 0x7fa9fc134de0 con 0x7fa9fc1430c0 2024-03-25T18:45:55.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.242+0000 7fa9f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:45:55.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.242+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:55.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.242+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:55.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.242+0000 7fa9f8ff9640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:45:55.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.242+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc1d7500 con 0x7fa9fc1430c0 2024-03-25T18:45:55.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.249+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3949329697 0 0) 0x7fa9fc1397d0 con 0x7fa9fc13f710 2024-03-25T18:45:55.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.249+0000 7fa9f8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:45:55.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.249+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:45:55.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.249+0000 7fa9f8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:45:55.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.249+0000 7fa9f8ff9640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:45:55.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.249+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa9fc134de0 con 0x7fa9fc13f710 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.255+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3147128577 0 0) 0x7fa9e8004eb0 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.255+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9fc1397d0 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.255+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa9ec001a30 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.255+0000 7fa9f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.255+0000 7fa9f8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.255+0000 7fa9f8ff9640 10 monclient(hunting): dump: 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.256 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:55.257 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:55.257 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:55.257 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:55.257 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:55.257 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:55.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.255+0000 7fa9f8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:45:55.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.259+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1275004922 0 0) 0x7fa9fc1d7500 con 0x7fa9fc1430c0 2024-03-25T18:45:55.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.259+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa9e8004eb0 con 0x7fa9fc1430c0 2024-03-25T18:45:55.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.259+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa9f4001b10 con 0x7fa9fc1430c0 2024-03-25T18:45:55.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.259+0000 7fa9f8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:55.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.259+0000 7fa9f8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.259+0000 7fa9f8ff9640 10 monclient(hunting): dump: 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:55.261 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1662330499 0 0) 0x7fa9fc1397d0 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 >> v1:172.21.15.32:6790/0 conn(0x7fa9fc1430c0 legacy=0x7fa9fc1e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 >> v1:172.21.15.32:6789/0 conn(0x7fa9fc13f710 legacy=0x7fa9fc1d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 1 monclient: found mon.b 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9fc134f70 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.279 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7fa9f8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7faa0230e640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7faa0230e640 10 monclient: _renew_subs 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7faa0230e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.277+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9fc138b30 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.278+0000 7faa0230e640 10 client.4516.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.278+0000 7faa0230e640 10 monclient: _renew_subs 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.278+0000 7faa0230e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.278+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9fc137880 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.278+0000 7faa0230e640 1 librados: init done 2024-03-25T18:45:55.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.279+0000 7faa0230e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:55.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.279+0000 7faa0230e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:55.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.279+0000 7faa0230e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:55.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.279+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9fc0627b0 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.582+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa9ec001c70 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.582+0000 7fa9f8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:45:55.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.582+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa9ec0038c0 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.582+0000 7fa9f8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:45:55.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.582+0000 7fa9f8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:45:55.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.582+0000 7fa9f8ff9640 10 monclient: dump: 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:45:55.584 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.582+0000 7fa9fa7fc640 4 set_mon_vals no callback set 2024-03-25T18:45:55.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.582+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fa9ec003aa0 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.583+0000 7fa9fb7fe640 10 monclient: get_auth_request con 0x7fa9e806df20 auth_method 0 2024-03-25T18:45:55.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.583+0000 7fa9f8ff9640 10 client.4516.objecter ms_handle_connect 0x7fa9e806df20 2024-03-25T18:45:55.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.588+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fa9ec054770 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.588+0000 7fa9f8ff9640 10 client.4516.objecter ms_dispatch 0x7fa9fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:45:55.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.588+0000 7fa9f8ff9640 3 client.4516.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:45:55.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.588+0000 7fa9f8ff9640 3 client.4516.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:45:55.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.588+0000 7fa9f8ff9640 20 client.4516.objecter dump_active .. 0 homeless 2024-03-25T18:45:55.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.588+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa9ec050280 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.588+0000 7fa9f8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:45:55.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.588+0000 7fa9f8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:45:55.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.841+0000 7faa0230e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:55.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.841+0000 7faa0230e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:55.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.841+0000 7faa0230e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:45:55.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.841+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa9fc069f40 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.873+0000 7fa9f8ff9640 1 -- 172.21.15.32:0/2892066089 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fa9ec0521b0 con 0x7fa9fc0a98f0 2024-03-25T18:45:55.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.873+0000 7fa9f8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:45:55.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.873+0000 7fa9f8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:45:55.875 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:45:55.875 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:45:55.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.875+0000 7faa0230e640 10 librados: watch_flush enter 2024-03-25T18:45:55.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 10 librados: watch_flush exit 2024-03-25T18:45:55.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 20 client.4516.objecter shutdown clearing up homeless session... 2024-03-25T18:45:55.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 10 client.4516.objecter successfully canceled tick 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 >> v1:172.21.15.136:6800/1141502808 conn(0x7fa9e806df20 legacy=0x7fa9e80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 10 monclient: shutdown 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 >> v1:172.21.15.136:6789/0 conn(0x7fa9fc0a98f0 legacy=0x7fa9fc138420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 shutdown_connections 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 >> 172.21.15.32:0/2892066089 conn(0x7fa9fc0b7ea0 msgr2=0x7fa9fc13c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 shutdown_connections 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 1 -- 172.21.15.32:0/2892066089 wait complete. 2024-03-25T18:45:55.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:45:55.876+0000 7faa0230e640 1 librados: shutdown 2024-03-25T18:45:55.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:46:02.697 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:46:02.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.812+0000 7fb6c5406640 10 monclient: get_monmap_and_config 2024-03-25T18:46:02.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.812+0000 7fb6c5406640 10 monclient: build_initial_monmap 2024-03-25T18:46:02.814 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.813+0000 7fb6c5406640 10 monclient: monmap: 2024-03-25T18:46:02.814 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:02.814 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:02.814 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:46:02.814476+0000 2024-03-25T18:46:02.815 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:46:02.814476+0000 2024-03-25T18:46:02.815 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:02.815 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:02.815 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:02.815 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:46:02.815 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:46:02.815 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:02.816 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.813+0000 7fb6c5406640 1 Processor -- start 2024-03-25T18:46:02.816 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.813+0000 7fb6c5406640 1 -- start start 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.813+0000 7fb6c5406640 10 monclient: init 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.814+0000 7fb6c5406640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.814+0000 7fb6c5406640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.814+0000 7fb6c5406640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c00a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.814+0000 7fb6c5406640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.814+0000 7fb6c5406640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c01430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.814+0000 7fb6c5406640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c0142520 con 0x7fb6c00a98f0 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.815+0000 7fb6c5406640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c00b28a0 con 0x7fb6c013f710 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.815+0000 7fb6c5406640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c00b2a30 con 0x7fb6c01430c0 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.815+0000 7fb6c5406640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.815+0000 7fb6c5406640 10 monclient(hunting): authenticate will time out at 6469.004571s 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.815+0000 7fb6b7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb6c013f710 0x7fb6c01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:46398/0 (socket says 172.21.15.32:46398) 2024-03-25T18:46:02.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.815+0000 7fb6b7fff640 1 -- 172.21.15.32:0/4107862744 learned_addr learned my addr 172.21.15.32:0/4107862744 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:02.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.832+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1382976203 0 0) 0x7fb6c00b28a0 con 0x7fb6c013f710 2024-03-25T18:46:02.834 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.832+0000 7fb6be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:46:02.834 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.832+0000 7fb6be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:02.834 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.832+0000 7fb6be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:02.834 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.832+0000 7fb6be7fc640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:46:02.834 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.832+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6ac003520 con 0x7fb6c013f710 2024-03-25T18:46:02.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.841+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491738135 0 0) 0x7fb6c00b2a30 con 0x7fb6c01430c0 2024-03-25T18:46:02.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.841+0000 7fb6be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:46:02.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.841+0000 7fb6be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:02.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.841+0000 7fb6be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:02.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.841+0000 7fb6be7fc640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:46:02.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.841+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c00b28a0 con 0x7fb6c01430c0 2024-03-25T18:46:02.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.845+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897138985 0 0) 0x7fb6c0142520 con 0x7fb6c00a98f0 2024-03-25T18:46:02.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.845+0000 7fb6be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:46:02.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.845+0000 7fb6be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:02.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.845+0000 7fb6be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:02.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.845+0000 7fb6be7fc640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:46:02.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.845+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c00b2a30 con 0x7fb6c00a98f0 2024-03-25T18:46:02.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3683547239 0 0) 0x7fb6ac003520 con 0x7fb6c013f710 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 >> v1:172.21.15.32:6790/0 conn(0x7fb6c01430c0 legacy=0x7fb6c014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 >> v1:172.21.15.32:6789/0 conn(0x7fb6c00a98f0 legacy=0x7fb6c00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 1 monclient: found mon.noname-c 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c00b44c0 con 0x7fb6c013f710 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb6a8001690 con 0x7fb6c013f710 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6c5406640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6c5406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:02.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6be7fc640 10 monclient: dump: 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:02.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.847+0000 7fb6c5406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:02.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.861+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6a80017f0 con 0x7fb6c013f710 2024-03-25T18:46:02.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.861+0000 7fb6be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.861+0000 7fb6be7fc640 1 -- 172.21.15.32:0/4107862744 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb6a8001c00 con 0x7fb6c013f710 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.861+0000 7fb6c5406640 10 monclient: get_monmap_and_config success 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.861+0000 7fb6c5406640 4 set_mon_vals no callback set 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.861+0000 7fb6be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.861+0000 7fb6be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.861+0000 7fb6be7fc640 10 monclient: dump: 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:02.863 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:02.864 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:02.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.862+0000 7fb6c5406640 10 monclient: shutdown 2024-03-25T18:46:02.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.862+0000 7fb6c5406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.862+0000 7fb6c5406640 1 -- 172.21.15.32:0/4107862744 >> v1:172.21.15.136:6789/0 conn(0x7fb6c013f710 legacy=0x7fb6c01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.862+0000 7fb6c5406640 1 -- 172.21.15.32:0/4107862744 shutdown_connections 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.862+0000 7fb6c5406640 1 -- 172.21.15.32:0/4107862744 >> 172.21.15.32:0/4107862744 conn(0x7fb6c00b7ea0 msgr2=0x7fb6c00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.862+0000 7fb6c5406640 1 -- 172.21.15.32:0/4107862744 shutdown_connections 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.862+0000 7fb6c5406640 1 -- 172.21.15.32:0/4107862744 wait complete. 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 10 monclient: build_initial_monmap 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 10 monclient: monmap: 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:02.866 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 1 librados: starting msgr at 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 1 librados: starting objecter 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 1 Processor -- start 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 1 -- start start 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 1 librados: setting wanted keys 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 1 librados: calling monclient init 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.863+0000 7fb6c5406640 10 monclient: init 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c00a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:02.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:02.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c01430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:02.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c0138a50 con 0x7fb6c00a98f0 2024-03-25T18:46:02.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c00ab9e0 con 0x7fb6c013f710 2024-03-25T18:46:02.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c01d7880 con 0x7fb6c01430c0 2024-03-25T18:46:02.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:02.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.864+0000 7fb6c5406640 10 monclient(hunting): authenticate will time out at 6469.054043s 2024-03-25T18:46:02.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.870+0000 7fb6b7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb6c013f710 0x7fb6c01d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:46434/0 (socket says 172.21.15.32:46434) 2024-03-25T18:46:02.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.870+0000 7fb6b7fff640 1 -- 172.21.15.32:0/361315672 learned_addr learned my addr 172.21.15.32:0/361315672 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:02.872 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.871+0000 7fb6b77fe640 10 client.?.objecter ms_handle_connect 0x7fb6c013f710 2024-03-25T18:46:02.872 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.871+0000 7fb6b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:02.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.877+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336251545 0 0) 0x7fb6c00ab9e0 con 0x7fb6c013f710 2024-03-25T18:46:02.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.877+0000 7fb6b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:46:02.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.877+0000 7fb6b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:02.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.877+0000 7fb6b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:02.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.877+0000 7fb6b77fe640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:46:02.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.877+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a0004eb0 con 0x7fb6c013f710 2024-03-25T18:46:02.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.881+0000 7fb6b77fe640 10 client.?.objecter ms_handle_connect 0x7fb6c00a98f0 2024-03-25T18:46:02.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.881+0000 7fb6b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:02.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.886+0000 7fb6b77fe640 10 client.?.objecter ms_handle_connect 0x7fb6c01430c0 2024-03-25T18:46:02.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.886+0000 7fb6b77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:02.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.893+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3528614721 0 0) 0x7fb6c0138a50 con 0x7fb6c00a98f0 2024-03-25T18:46:02.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.893+0000 7fb6b77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:46:02.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.893+0000 7fb6b77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:02.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.893+0000 7fb6b77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:02.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.893+0000 7fb6b77fe640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:46:02.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.893+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c00ab9e0 con 0x7fb6c00a98f0 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.894+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 457754453 0 0) 0x7fb6a0004eb0 con 0x7fb6c013f710 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.894+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6c0138a50 con 0x7fb6c013f710 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.894+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb6a8001bc0 con 0x7fb6c013f710 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.894+0000 7fb6b77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.894+0000 7fb6b77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.894+0000 7fb6b77fe640 10 monclient(hunting): dump: 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:02.896 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:02.897 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:02.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.894+0000 7fb6b77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2320782715 0 0) 0x7fb6c0138a50 con 0x7fb6c013f710 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 >> v1:172.21.15.32:6790/0 conn(0x7fb6c01430c0 legacy=0x7fb6c01e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 >> v1:172.21.15.32:6789/0 conn(0x7fb6c00a98f0 legacy=0x7fb6c0138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 1 monclient: found mon.b 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c01d84b0 con 0x7fb6c013f710 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 10 monclient: _finish_auth 0 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6b77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:02.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:46:02.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 10 monclient: _renew_subs 2024-03-25T18:46:02.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:02.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c01d51f0 con 0x7fb6c013f710 2024-03-25T18:46:02.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 10 client.4522.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:46:02.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 10 monclient: _renew_subs 2024-03-25T18:46:02.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:02.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c0138e60 con 0x7fb6c013f710 2024-03-25T18:46:02.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.901+0000 7fb6c5406640 1 librados: init done 2024-03-25T18:46:02.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.902+0000 7fb6c5406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:02.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.902+0000 7fb6c5406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:02.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.902+0000 7fb6c5406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:02.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.902+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c01d76e0 con 0x7fb6c013f710 2024-03-25T18:46:02.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.905+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6a80011f0 con 0x7fb6c013f710 2024-03-25T18:46:02.907 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.905+0000 7fb6b77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:02.907 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:02.905+0000 7fb6bdffb640 4 set_mon_vals no callback set 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.171+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb6a8004b10 con 0x7fb6c013f710 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.171+0000 7fb6b77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.171+0000 7fb6b77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.171+0000 7fb6b77fe640 10 monclient: dump: 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:03.173 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:03.174 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:03.174 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:03.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.172+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fb6a801aba0 con 0x7fb6c013f710 2024-03-25T18:46:03.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.173+0000 7fb6beffd640 10 monclient: get_auth_request con 0x7fb6a006dc70 auth_method 0 2024-03-25T18:46:03.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.173+0000 7fb6b77fe640 10 client.4522.objecter ms_handle_connect 0x7fb6a006dc70 2024-03-25T18:46:03.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.175+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fb6a80548b0 con 0x7fb6c013f710 2024-03-25T18:46:03.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.175+0000 7fb6b77fe640 10 client.4522.objecter ms_dispatch 0x7fb6c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:46:03.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.175+0000 7fb6b77fe640 3 client.4522.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:46:03.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.175+0000 7fb6b77fe640 3 client.4522.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:46:03.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.175+0000 7fb6b77fe640 20 client.4522.objecter dump_active .. 0 homeless 2024-03-25T18:46:03.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.177+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6a80501f0 con 0x7fb6c013f710 2024-03-25T18:46:03.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.177+0000 7fb6b77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:03.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.177+0000 7fb6b77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:46:03.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.428+0000 7fb6c5406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:03.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.428+0000 7fb6c5406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:03.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.428+0000 7fb6c5406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:03.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.428+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6c00627b0 con 0x7fb6c013f710 2024-03-25T18:46:03.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.453+0000 7fb6b77fe640 1 -- 172.21.15.32:0/361315672 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fb6a80522f0 con 0x7fb6c013f710 2024-03-25T18:46:03.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.453+0000 7fb6b77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:03.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.453+0000 7fb6b77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:46:03.454 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:46:03.455 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:46:03.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 10 librados: watch_flush enter 2024-03-25T18:46:03.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 10 librados: watch_flush exit 2024-03-25T18:46:03.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 20 client.4522.objecter shutdown clearing up homeless session... 2024-03-25T18:46:03.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 10 client.4522.objecter successfully canceled tick 2024-03-25T18:46:03.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 >> v1:172.21.15.136:6800/1141502808 conn(0x7fb6a006dc70 legacy=0x7fb6a0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:03.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 10 monclient: shutdown 2024-03-25T18:46:03.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:03.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 >> v1:172.21.15.136:6789/0 conn(0x7fb6c013f710 legacy=0x7fb6c01d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:03.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 shutdown_connections 2024-03-25T18:46:03.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 >> 172.21.15.32:0/361315672 conn(0x7fb6c00b7ea0 msgr2=0x7fb6c0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:03.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 shutdown_connections 2024-03-25T18:46:03.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 1 -- 172.21.15.32:0/361315672 wait complete. 2024-03-25T18:46:03.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:03.455+0000 7fb6c5406640 1 librados: shutdown 2024-03-25T18:46:03.468 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:46:08.243 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:20.571 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:08.252 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:20.580 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:08.281 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:20.609 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:08.302 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:20.618 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:08.308 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:20.625 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:08.330 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:20.655 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:08.336 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:20.664 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:08.341 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:20.657 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:08.370 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:20.695 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:10.276 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:46:10.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.376+0000 7f06a4f8e640 10 monclient: get_monmap_and_config 2024-03-25T18:46:10.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.376+0000 7f06a4f8e640 10 monclient: build_initial_monmap 2024-03-25T18:46:10.378 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.376+0000 7f06a4f8e640 10 monclient: monmap: 2024-03-25T18:46:10.378 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:10.378 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:10.378 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:46:10.378370+0000 2024-03-25T18:46:10.378 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:46:10.378370+0000 2024-03-25T18:46:10.379 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:10.379 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:10.379 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:10.379 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:46:10.379 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:46:10.379 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:10.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.377+0000 7f06a4f8e640 1 Processor -- start 2024-03-25T18:46:10.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.377+0000 7f06a4f8e640 1 -- start start 2024-03-25T18:46:10.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.377+0000 7f06a4f8e640 10 monclient: init 2024-03-25T18:46:10.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:10.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 10 monclient(hunting): picked mon.noname-c con 0x7f06a00a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 10 monclient(hunting): picked mon.noname-b con 0x7f06a013f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 10 monclient(hunting): picked mon.noname-a con 0x7f06a01430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0142520 con 0x7f06a01430c0 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a00b28a0 con 0x7f06a00a98f0 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a00b2a30 con 0x7f06a013f710 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f06a4f8e640 10 monclient(hunting): authenticate will time out at 6476.567980s 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.378+0000 7f0697fff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f06a013f710 0x7f06a01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:50602/0 (socket says 172.21.15.32:50602) 2024-03-25T18:46:10.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.379+0000 7f0697fff640 1 -- 172.21.15.32:0/2017767525 learned_addr learned my addr 172.21.15.32:0/2017767525 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:10.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.439+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1907629640 0 0) 0x7f06a0142520 con 0x7f06a01430c0 2024-03-25T18:46:10.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.439+0000 7f069dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:46:10.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.439+0000 7f069dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:10.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.439+0000 7f069dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:10.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.439+0000 7f069dd9b640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:46:10.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.439+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f068c003520 con 0x7f06a01430c0 2024-03-25T18:46:10.446 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.445+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 665591770 0 0) 0x7f06a00b2a30 con 0x7f06a013f710 2024-03-25T18:46:10.446 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.445+0000 7f069dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:46:10.447 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.445+0000 7f069dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:10.447 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.445+0000 7f069dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:10.447 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.445+0000 7f069dd9b640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:46:10.447 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.445+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a0142520 con 0x7f06a013f710 2024-03-25T18:46:10.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2879693418 0 0) 0x7f068c003520 con 0x7f06a01430c0 2024-03-25T18:46:10.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 >> v1:172.21.15.32:6790/0 conn(0x7f06a013f710 legacy=0x7f06a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:10.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 >> v1:172.21.15.136:6789/0 conn(0x7f06a00a98f0 legacy=0x7f06a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:10.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 1 monclient: found mon.noname-a 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00b44c0 con 0x7f06a01430c0 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f06a4f8e640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0698001a10 con 0x7f06a01430c0 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f069dd9b640 10 monclient: dump: 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:10.467 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:10.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.465+0000 7f06a4f8e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:10.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.549+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0698001b70 con 0x7f06a01430c0 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.549+0000 7f069dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.549+0000 7f069dd9b640 1 -- 172.21.15.32:0/2017767525 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0698001f80 con 0x7f06a01430c0 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.549+0000 7f06a4f8e640 10 monclient: get_monmap_and_config success 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.549+0000 7f06a4f8e640 4 set_mon_vals no callback set 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.549+0000 7f069dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.549+0000 7f069dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.549+0000 7f069dd9b640 10 monclient: dump: 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:10.551 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:10.552 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:10.552 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:10.552 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:10.552 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:10.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.550+0000 7f06a4f8e640 10 monclient: shutdown 2024-03-25T18:46:10.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.550+0000 7f06a4f8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:10.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.550+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2017767525 >> v1:172.21.15.32:6789/0 conn(0x7f06a01430c0 legacy=0x7f06a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:10.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.550+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2017767525 shutdown_connections 2024-03-25T18:46:10.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.550+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2017767525 >> 172.21.15.32:0/2017767525 conn(0x7f06a00b7ea0 msgr2=0x7f06a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:10.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.550+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2017767525 shutdown_connections 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.550+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2017767525 wait complete. 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.551+0000 7f06a4f8e640 10 monclient: build_initial_monmap 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.551+0000 7f06a4f8e640 10 monclient: monmap: 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 librados: starting msgr at 2024-03-25T18:46:10.555 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 librados: starting objecter 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 Processor -- start 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 -- start start 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 librados: setting wanted keys 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 librados: calling monclient init 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 10 monclient: init 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 10 monclient(hunting): picked mon.noname-a con 0x7f06a00a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 10 monclient(hunting): picked mon.noname-b con 0x7f06a013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 10 monclient(hunting): picked mon.noname-c con 0x7f06a01430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a013b410 con 0x7f06a00a98f0 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a013b010 con 0x7f06a013f710 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0135230 con 0x7f06a01430c0 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:10.556 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.552+0000 7f06a4f8e640 10 monclient(hunting): authenticate will time out at 6476.742515s 2024-03-25T18:46:10.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.572+0000 7f0697fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f06a013f710 0x7f06a01dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:37932/0 (socket says 172.21.15.32:37932) 2024-03-25T18:46:10.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.572+0000 7f0697fff640 1 -- 172.21.15.32:0/2810342765 learned_addr learned my addr 172.21.15.32:0/2810342765 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:10.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.576+0000 7f0696ffd640 10 client.?.objecter ms_handle_connect 0x7f06a013f710 2024-03-25T18:46:10.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.576+0000 7f0696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:10.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.582+0000 7f0696ffd640 10 client.?.objecter ms_handle_connect 0x7f06a00a98f0 2024-03-25T18:46:10.583 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.582+0000 7f0696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871633320 0 0) 0x7f06a013b010 con 0x7f06a013f710 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0680004eb0 con 0x7f06a013f710 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4067947081 0 0) 0x7f06a013b410 con 0x7f06a00a98f0 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:10.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:10.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:46:10.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.586+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f06a013b010 con 0x7f06a00a98f0 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.603+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 919393706 0 0) 0x7f0680004eb0 con 0x7f06a013f710 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.603+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f06a013b410 con 0x7f06a013f710 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.603+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f06880019b0 con 0x7f06a013f710 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.603+0000 7f0696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.603+0000 7f0696ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.603+0000 7f0696ffd640 10 monclient(hunting): dump: 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:10.605 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.606 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.606 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:10.606 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:10.606 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:10.606 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:10.606 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:10.606 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:10.606 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.603+0000 7f0696ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.605+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090913748 0 0) 0x7f06a013b010 con 0x7f06a00a98f0 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.605+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0680004eb0 con 0x7f06a00a98f0 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.605+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0690001930 con 0x7f06a00a98f0 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.605+0000 7f0696ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.605+0000 7f0696ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.605+0000 7f0696ffd640 10 monclient(hunting): dump: 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:10.607 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:10.608 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:10.608 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:10.608 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:10.608 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:10.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.615+0000 7f0696ffd640 10 client.?.objecter ms_handle_connect 0x7f06a01430c0 2024-03-25T18:46:10.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.615+0000 7f0696ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4189150095 0 0) 0x7f06a013b410 con 0x7f06a013f710 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 >> v1:172.21.15.32:6790/0 conn(0x7f06a01430c0 legacy=0x7f06a01db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 >> v1:172.21.15.32:6789/0 conn(0x7f06a00a98f0 legacy=0x7f06a0130ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 1 monclient: found mon.b 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a01353c0 con 0x7f06a013f710 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 10 monclient: _finish_auth 0 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f0696ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:10.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:46:10.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 10 monclient: _renew_subs 2024-03-25T18:46:10.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:10.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a00b78e0 con 0x7f06a013f710 2024-03-25T18:46:10.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:46:10.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 10 monclient: _renew_subs 2024-03-25T18:46:10.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:10.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a01317d0 con 0x7f06a013f710 2024-03-25T18:46:10.626 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.623+0000 7f06a4f8e640 1 librados: init done 2024-03-25T18:46:10.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.624+0000 7f06a4f8e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:10.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.624+0000 7f06a4f8e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:10.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.624+0000 7f06a4f8e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:10.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.624+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a00b7c80 con 0x7f06a013f710 2024-03-25T18:46:10.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.920+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f06880047e0 con 0x7f06a013f710 2024-03-25T18:46:10.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.920+0000 7f0696ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:10.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.920+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0688004af0 con 0x7f06a013f710 2024-03-25T18:46:10.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.920+0000 7f0696ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:10.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.920+0000 7f0696ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:10.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.920+0000 7f0696ffd640 10 monclient: dump: 2024-03-25T18:46:10.922 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:10.922 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:10.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.920+0000 7f069d59a640 4 set_mon_vals no callback set 2024-03-25T18:46:10.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.922+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f068801ab00 con 0x7f06a013f710 2024-03-25T18:46:10.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.922+0000 7f069e59c640 10 monclient: get_auth_request con 0x7f068006de80 auth_method 0 2024-03-25T18:46:10.924 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.922+0000 7f0696ffd640 10 client.4525.objecter ms_handle_connect 0x7f068006de80 2024-03-25T18:46:10.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.925+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f0688054870 con 0x7f06a013f710 2024-03-25T18:46:10.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.925+0000 7f0696ffd640 10 client.4525.objecter ms_dispatch 0x7f06a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:46:10.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.925+0000 7f0696ffd640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:46:10.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.925+0000 7f0696ffd640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:46:10.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.926+0000 7f0696ffd640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T18:46:10.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.926+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f06880501b0 con 0x7f06a013f710 2024-03-25T18:46:10.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.926+0000 7f0696ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:10.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:10.926+0000 7f0696ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:46:11.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.177+0000 7f06a4f8e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:11.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.177+0000 7f06a4f8e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:11.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.177+0000 7f06a4f8e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:11.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.177+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f06a013b1a0 con 0x7f06a013f710 2024-03-25T18:46:11.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.208+0000 7f0696ffd640 1 -- 172.21.15.32:0/2810342765 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f06880522b0 con 0x7f06a013f710 2024-03-25T18:46:11.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.208+0000 7f0696ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:11.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.208+0000 7f0696ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:46:11.210 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:46:11.210 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:46:11.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.210+0000 7f06a4f8e640 10 librados: watch_flush enter 2024-03-25T18:46:11.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.210+0000 7f06a4f8e640 10 librados: watch_flush exit 2024-03-25T18:46:11.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.210+0000 7f06a4f8e640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T18:46:11.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.210+0000 7f06a4f8e640 10 client.4525.objecter successfully canceled tick 2024-03-25T18:46:11.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.210+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 >> v1:172.21.15.136:6800/1141502808 conn(0x7f068006de80 legacy=0x7f0680070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:11.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.211+0000 7f06a4f8e640 10 monclient: shutdown 2024-03-25T18:46:11.212 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.211+0000 7f06a4f8e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:11.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.211+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 >> v1:172.21.15.136:6789/0 conn(0x7f06a013f710 legacy=0x7f06a01dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:11.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.211+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 shutdown_connections 2024-03-25T18:46:11.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.211+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 >> 172.21.15.32:0/2810342765 conn(0x7f06a00b7ea0 msgr2=0x7f06a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:11.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.211+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 shutdown_connections 2024-03-25T18:46:11.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.211+0000 7f06a4f8e640 1 -- 172.21.15.32:0/2810342765 wait complete. 2024-03-25T18:46:11.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:11.211+0000 7f06a4f8e640 1 librados: shutdown 2024-03-25T18:46:11.224 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:46:18.031 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:46:18.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.149+0000 7fa104686640 10 monclient: get_monmap_and_config 2024-03-25T18:46:18.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.149+0000 7fa104686640 10 monclient: build_initial_monmap 2024-03-25T18:46:18.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.149+0000 7fa104686640 10 monclient: monmap: 2024-03-25T18:46:18.150 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:18.150 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:18.150 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:46:18.150294+0000 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:46:18.150294+0000 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.150+0000 7fa104686640 1 Processor -- start 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.150+0000 7fa104686640 1 -- start start 2024-03-25T18:46:18.151 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.150+0000 7fa104686640 10 monclient: init 2024-03-25T18:46:18.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:18.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:46:18.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 10 monclient(hunting): picked mon.noname-b con 0x7fa0fc0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:18.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 10 monclient(hunting): picked mon.noname-a con 0x7fa0fc13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:18.152 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 10 monclient(hunting): picked mon.noname-c con 0x7fa0fc1430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:18.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc142520 con 0x7fa0fc13f710 2024-03-25T18:46:18.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc0b28a0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc0b2a30 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:18.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.151+0000 7fa104686640 10 monclient(hunting): authenticate will time out at 6484.340222s 2024-03-25T18:46:18.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.163+0000 7fa101c21640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0fc13f710 0x7fa0fc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56736/0 (socket says 172.21.15.32:56736) 2024-03-25T18:46:18.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.163+0000 7fa101c21640 1 -- 172.21.15.32:0/3055557732 learned_addr learned my addr 172.21.15.32:0/3055557732 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:18.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.178+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1046990839 0 0) 0x7fa0fc142520 con 0x7fa0fc13f710 2024-03-25T18:46:18.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.179+0000 7fa101420640 10 monclient(hunting): _negotiate 2024-03-25T18:46:18.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.179+0000 7fa101420640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:18.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.179+0000 7fa101420640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:18.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.179+0000 7fa101420640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:46:18.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.179+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e8003520 con 0x7fa0fc13f710 2024-03-25T18:46:18.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.189+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260452315 0 0) 0x7fa0fc0b28a0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.189+0000 7fa101420640 10 monclient(hunting): _negotiate 2024-03-25T18:46:18.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.189+0000 7fa101420640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:18.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.189+0000 7fa101420640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:18.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.189+0000 7fa101420640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:46:18.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.189+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc142520 con 0x7fa0fc1430c0 2024-03-25T18:46:18.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.192+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660921822 0 0) 0x7fa0fc0b2a30 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.192+0000 7fa101420640 10 monclient(hunting): _negotiate 2024-03-25T18:46:18.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.192+0000 7fa101420640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:18.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.192+0000 7fa101420640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:18.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.192+0000 7fa101420640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:46:18.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.192+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc0b28a0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3663105915 0 0) 0x7fa0fc142520 con 0x7fa0fc1430c0 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 >> v1:172.21.15.32:6790/0 conn(0x7fa0fc0a98f0 legacy=0x7fa0fc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 >> v1:172.21.15.32:6789/0 conn(0x7fa0fc13f710 legacy=0x7fa0fc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 10 monclient: _finish_hunting 0 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 1 monclient: found mon.noname-c 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc0b44c0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 10 monclient: _finish_auth 0 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 10 monclient: _check_auth_tickets 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa0f40019f0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa101420640 10 monclient: dump: 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa104686640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:46:18.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.196+0000 7fa104686640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:18.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.214+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f4001b50 con 0x7fa0fc1430c0 2024-03-25T18:46:18.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.214+0000 7fa101420640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.214+0000 7fa101420640 1 -- 172.21.15.32:0/3055557732 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa0f4001f60 con 0x7fa0fc1430c0 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.214+0000 7fa101420640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.214+0000 7fa101420640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.214+0000 7fa101420640 10 monclient: dump: 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:18.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.214+0000 7fa104686640 10 monclient: get_monmap_and_config success 2024-03-25T18:46:18.216 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.214+0000 7fa104686640 4 set_mon_vals no callback set 2024-03-25T18:46:18.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.215+0000 7fa104686640 10 monclient: shutdown 2024-03-25T18:46:18.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.215+0000 7fa104686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:18.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.215+0000 7fa104686640 1 -- 172.21.15.32:0/3055557732 >> v1:172.21.15.136:6789/0 conn(0x7fa0fc1430c0 legacy=0x7fa0fc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:18.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.215+0000 7fa104686640 1 -- 172.21.15.32:0/3055557732 shutdown_connections 2024-03-25T18:46:18.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.215+0000 7fa104686640 1 -- 172.21.15.32:0/3055557732 >> 172.21.15.32:0/3055557732 conn(0x7fa0fc0b7ea0 msgr2=0x7fa0fc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:18.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.215+0000 7fa104686640 1 -- 172.21.15.32:0/3055557732 shutdown_connections 2024-03-25T18:46:18.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.215+0000 7fa104686640 1 -- 172.21.15.32:0/3055557732 wait complete. 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.216+0000 7fa104686640 10 monclient: build_initial_monmap 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.216+0000 7fa104686640 10 monclient: monmap: 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.217+0000 7fa104686640 1 librados: starting msgr at 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.217+0000 7fa104686640 1 librados: starting objecter 2024-03-25T18:46:18.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.217+0000 7fa104686640 1 Processor -- start 2024-03-25T18:46:18.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.217+0000 7fa104686640 1 -- start start 2024-03-25T18:46:18.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.217+0000 7fa104686640 1 librados: setting wanted keys 2024-03-25T18:46:18.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.217+0000 7fa104686640 1 librados: calling monclient init 2024-03-25T18:46:18.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.217+0000 7fa104686640 10 monclient: init 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 10 monclient(hunting): picked mon.noname-c con 0x7fa0fc0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 10 monclient(hunting): picked mon.noname-b con 0x7fa0fc13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 10 monclient(hunting): picked mon.noname-a con 0x7fa0fc1430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc1388a0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc1349e0 con 0x7fa0fc13f710 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0fc138d30 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:18.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.218+0000 7fa104686640 10 monclient(hunting): authenticate will time out at 6484.407020s 2024-03-25T18:46:18.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.229+0000 7fa102c23640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa0fc1430c0 0x7fa0fc1342d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:56774/0 (socket says 172.21.15.32:56774) 2024-03-25T18:46:18.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.229+0000 7fa102c23640 1 -- 172.21.15.32:0/656388846 learned_addr learned my addr 172.21.15.32:0/656388846 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:18.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.242+0000 7fa0f2ffd640 10 client.?.objecter ms_handle_connect 0x7fa0fc1430c0 2024-03-25T18:46:18.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.242+0000 7fa0f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:18.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.243+0000 7fa0f2ffd640 10 client.?.objecter ms_handle_connect 0x7fa0fc13f710 2024-03-25T18:46:18.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.243+0000 7fa0f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:18.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.245+0000 7fa0f2ffd640 10 client.?.objecter ms_handle_connect 0x7fa0fc0a98f0 2024-03-25T18:46:18.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.245+0000 7fa0f2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:18.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.248+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860172293 0 0) 0x7fa0fc1388a0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.249+0000 7fa0f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:18.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.249+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:18.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.249+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:18.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.249+0000 7fa0f2ffd640 10 monclient(hunting): my global_id is 4551 2024-03-25T18:46:18.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.249+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e0004eb0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.250+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523048208 0 0) 0x7fa0fc1349e0 con 0x7fa0fc13f710 2024-03-25T18:46:18.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.250+0000 7fa0f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:18.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.250+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:18.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.250+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:18.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.250+0000 7fa0f2ffd640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:46:18.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.250+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc1388a0 con 0x7fa0fc13f710 2024-03-25T18:46:18.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.257+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 564038658 0 0) 0x7fa0fc138d30 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.257+0000 7fa0f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:18.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.257+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:18.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.257+0000 7fa0f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:18.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.257+0000 7fa0f2ffd640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:46:18.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.257+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0fc1349e0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.263+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1275234393 0 0) 0x7fa0e0004eb0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.263+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0fc138d30 con 0x7fa0fc1430c0 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.263+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa0f4001fa0 con 0x7fa0fc1430c0 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.263+0000 7fa0f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.263+0000 7fa0f2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.263+0000 7fa0f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:18.264 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:18.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.263+0000 7fa0f2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:46:18.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.269+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4165983193 0 0) 0x7fa0fc1349e0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.269+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0e0004eb0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.269+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa0f8001830 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.269+0000 7fa0f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:18.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.269+0000 7fa0f2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:46:18.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.269+0000 7fa0f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:46:18.269 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:18.269 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:18.270 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.270 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.270 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:18.270 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:18.270 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:18.270 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:18.270 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:18.270 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.273+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1167533047 0 0) 0x7fa0fc1388a0 con 0x7fa0fc13f710 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.273+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0fc1349e0 con 0x7fa0fc13f710 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.273+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa0ec001950 con 0x7fa0fc13f710 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.273+0000 7fa0f2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.274+0000 7fa0f2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.274+0000 7fa0f2ffd640 10 monclient(hunting): dump: 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:18.274 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:18.275 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:18.275 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:18.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3432370040 0 0) 0x7fa0e0004eb0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 >> v1:172.21.15.136:6789/0 conn(0x7fa0fc13f710 legacy=0x7fa0fc133bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:18.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 >> v1:172.21.15.32:6789/0 conn(0x7fa0fc1430c0 legacy=0x7fa0fc1342d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:18.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 1 monclient: found mon.c 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0fc0aba50 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa0f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 5 monclient: authenticate success, global_id 4538 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 10 monclient: _renew_subs 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0fc0b1770 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 10 client.4538.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 10 monclient: _renew_subs 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:18.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0fc137ba0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.280+0000 7fa104686640 1 librados: init done 2024-03-25T18:46:18.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.281+0000 7fa104686640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:18.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.281+0000 7fa104686640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:18.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.281+0000 7fa104686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:18.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.281+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0fc0627b0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.609+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f8001d40 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.609+0000 7fa0f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:18.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.609+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa0f80037f0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.609+0000 7fa0f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:18.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.609+0000 7fa0f2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.609+0000 7fa0f2ffd640 10 monclient: dump: 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:18.611 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.609+0000 7fa100c1f640 4 set_mon_vals no callback set 2024-03-25T18:46:18.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.610+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fa0f8019880 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.611+0000 7fa101c21640 10 monclient: get_auth_request con 0x7fa0e006e160 auth_method 0 2024-03-25T18:46:18.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.611+0000 7fa0f2ffd640 10 client.4538.objecter ms_handle_connect 0x7fa0e006e160 2024-03-25T18:46:18.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.615+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fa0f8053560 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.615+0000 7fa0f2ffd640 10 client.4538.objecter ms_dispatch 0x7fa0fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:46:18.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.615+0000 7fa0f2ffd640 3 client.4538.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:46:18.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.615+0000 7fa0f2ffd640 3 client.4538.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:46:18.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.615+0000 7fa0f2ffd640 20 client.4538.objecter dump_active .. 0 homeless 2024-03-25T18:46:18.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.615+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0f804eea0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.615+0000 7fa0f2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:18.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.615+0000 7fa0f2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:46:18.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.870+0000 7fa104686640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:18.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.870+0000 7fa104686640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:18.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.870+0000 7fa104686640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:18.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.870+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0fc141a00 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.937 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.937+0000 7fa0f2ffd640 1 -- 172.21.15.32:0/656388846 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fa0f8050fa0 con 0x7fa0fc0a98f0 2024-03-25T18:46:18.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.937+0000 7fa0f2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:18.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.937+0000 7fa0f2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:46:18.951 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:46:18.952 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:46:18.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 10 librados: watch_flush enter 2024-03-25T18:46:18.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 10 librados: watch_flush exit 2024-03-25T18:46:18.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 20 client.4538.objecter shutdown clearing up homeless session... 2024-03-25T18:46:18.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 10 client.4538.objecter successfully canceled tick 2024-03-25T18:46:18.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 >> v1:172.21.15.136:6800/1141502808 conn(0x7fa0e006e160 legacy=0x7fa0e0070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:18.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 10 monclient: shutdown 2024-03-25T18:46:18.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:18.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 >> v1:172.21.15.32:6790/0 conn(0x7fa0fc0a98f0 legacy=0x7fa0fc1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:18.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 shutdown_connections 2024-03-25T18:46:18.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 >> 172.21.15.32:0/656388846 conn(0x7fa0fc0b7ea0 msgr2=0x7fa0fc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:18.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 shutdown_connections 2024-03-25T18:46:18.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.939+0000 7fa104686640 1 -- 172.21.15.32:0/656388846 wait complete. 2024-03-25T18:46:18.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:18.940+0000 7fa104686640 1 librados: shutdown 2024-03-25T18:46:18.954 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:46:25.762 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:46:25.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.878+0000 7f096f21e640 10 monclient: get_monmap_and_config 2024-03-25T18:46:25.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.878+0000 7f096f21e640 10 monclient: build_initial_monmap 2024-03-25T18:46:25.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.878+0000 7f096f21e640 10 monclient: monmap: 2024-03-25T18:46:25.879 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:25.879 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:25.879 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:46:25.879153+0000 2024-03-25T18:46:25.879 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:46:25.879153+0000 2024-03-25T18:46:25.880 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:25.880 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:25.880 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:25.880 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:46:25.880 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:46:25.880 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:25.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.879+0000 7f096f21e640 1 Processor -- start 2024-03-25T18:46:25.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.879+0000 7f096f21e640 1 -- start start 2024-03-25T18:46:25.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.879+0000 7f096f21e640 10 monclient: init 2024-03-25T18:46:25.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:25.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:46:25.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 10 monclient(hunting): picked mon.noname-b con 0x7f09680a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:25.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 10 monclient(hunting): picked mon.noname-c con 0x7f096813f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:25.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 10 monclient(hunting): picked mon.noname-a con 0x7f09681430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:25.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0968142520 con 0x7f09681430c0 2024-03-25T18:46:25.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680b28a0 con 0x7f096813f710 2024-03-25T18:46:25.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680b2a30 con 0x7f09680a98f0 2024-03-25T18:46:25.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:25.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.880+0000 7f096f21e640 10 monclient(hunting): authenticate will time out at 6492.069137s 2024-03-25T18:46:25.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.888+0000 7f0967fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f096813f710 0x7f09681429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:45516/0 (socket says 172.21.15.32:45516) 2024-03-25T18:46:25.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.888+0000 7f0967fff640 1 -- 172.21.15.32:0/2201329729 learned_addr learned my addr 172.21.15.32:0/2201329729 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:25.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.898+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3912696294 0 0) 0x7f09680b2a30 con 0x7f09680a98f0 2024-03-25T18:46:25.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.898+0000 7f09677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:46:25.899 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.898+0000 7f09677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:25.899 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.898+0000 7f09677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:25.899 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.898+0000 7f09677fe640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:46:25.899 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.898+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0958003520 con 0x7f09680a98f0 2024-03-25T18:46:25.899 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.899+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001057563 0 0) 0x7f09680b28a0 con 0x7f096813f710 2024-03-25T18:46:25.899 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.899+0000 7f09677fe640 10 monclient(hunting): _negotiate 2024-03-25T18:46:25.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.899+0000 7f09677fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:25.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.899+0000 7f09677fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:25.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.899+0000 7f09677fe640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:46:25.900 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.899+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09680b2a30 con 0x7f096813f710 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.910+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2437349738 0 0) 0x7f09680b2a30 con 0x7f096813f710 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 >> v1:172.21.15.32:6790/0 conn(0x7f09680a98f0 legacy=0x7f09680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 >> v1:172.21.15.32:6789/0 conn(0x7f09681430c0 legacy=0x7f096814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 10 monclient: _finish_hunting 0 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 1 monclient: found mon.noname-c 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09680b44c0 con 0x7f096813f710 2024-03-25T18:46:25.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 10 monclient: _finish_auth 0 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 10 monclient: _check_auth_tickets 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0954001670 con 0x7f096813f710 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f096f21e640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f096f21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f09677fe640 10 monclient: dump: 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:25.912 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:25.913 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:25.913 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:25.913 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.911+0000 7f096f21e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:25.925 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.925+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09540017d0 con 0x7f096813f710 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.925+0000 7f09677fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.925+0000 7f09677fe640 1 -- 172.21.15.32:0/2201329729 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0954001b60 con 0x7f096813f710 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.925+0000 7f09677fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.925+0000 7f09677fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.925+0000 7f09677fe640 10 monclient: dump: 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:25.926 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:25.927 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:25.927 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:25.927 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:25.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.925+0000 7f096f21e640 10 monclient: get_monmap_and_config success 2024-03-25T18:46:25.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.925+0000 7f096f21e640 4 set_mon_vals no callback set 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.926+0000 7f096f21e640 10 monclient: shutdown 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.926+0000 7f096f21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.926+0000 7f096f21e640 1 -- 172.21.15.32:0/2201329729 >> v1:172.21.15.136:6789/0 conn(0x7f096813f710 legacy=0x7f09681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.926+0000 7f096f21e640 1 -- 172.21.15.32:0/2201329729 shutdown_connections 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.926+0000 7f096f21e640 1 -- 172.21.15.32:0/2201329729 >> 172.21.15.32:0/2201329729 conn(0x7f09680b7ea0 msgr2=0x7f09680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.926+0000 7f096f21e640 1 -- 172.21.15.32:0/2201329729 shutdown_connections 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.927+0000 7f096f21e640 1 -- 172.21.15.32:0/2201329729 wait complete. 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.927+0000 7f096f21e640 10 monclient: build_initial_monmap 2024-03-25T18:46:25.929 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.927+0000 7f096f21e640 10 monclient: monmap: 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 1 librados: starting msgr at 2024-03-25T18:46:25.930 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 1 librados: starting objecter 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 1 Processor -- start 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 1 -- start start 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 1 librados: setting wanted keys 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 1 librados: calling monclient init 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 10 monclient: init 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 10 monclient(hunting): picked mon.noname-c con 0x7f09680a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.928+0000 7f096f21e640 10 monclient(hunting): picked mon.noname-b con 0x7f096813f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.929+0000 7f096f21e640 10 monclient(hunting): picked mon.noname-a con 0x7f09681430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.929+0000 7f096f21e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f096813b720 con 0x7f09681430c0 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.929+0000 7f096f21e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0968139d20 con 0x7f096813f710 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.929+0000 7f096f21e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09680b7b70 con 0x7f09680a98f0 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.929+0000 7f096f21e640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:25.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.929+0000 7f096f21e640 10 monclient(hunting): authenticate will time out at 6492.117644s 2024-03-25T18:46:25.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.932+0000 7f0967fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f096813f710 0x7f09681df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:45548/0 (socket says 172.21.15.32:45548) 2024-03-25T18:46:25.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.932+0000 7f0967fff640 1 -- 172.21.15.32:0/335019718 learned_addr learned my addr 172.21.15.32:0/335019718 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:25.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.934+0000 7f09657fa640 10 client.?.objecter ms_handle_connect 0x7f096813f710 2024-03-25T18:46:25.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.934+0000 7f09657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:25.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.940+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838680170 0 0) 0x7f0968139d20 con 0x7f096813f710 2024-03-25T18:46:25.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.940+0000 7f09657fa640 10 monclient(hunting): _negotiate 2024-03-25T18:46:25.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.940+0000 7f09657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:25.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.940+0000 7f09657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:25.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.940+0000 7f09657fa640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:46:25.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.940+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f094c004eb0 con 0x7f096813f710 2024-03-25T18:46:25.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.941+0000 7f09657fa640 10 client.?.objecter ms_handle_connect 0x7f09681430c0 2024-03-25T18:46:25.941 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.941+0000 7f09657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:25.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.947+0000 7f09657fa640 10 client.?.objecter ms_handle_connect 0x7f09680a98f0 2024-03-25T18:46:25.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.947+0000 7f09657fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.952+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1810526616 0 0) 0x7f094c004eb0 con 0x7f096813f710 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0968139d20 con 0x7f096813f710 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0954001ba0 con 0x7f096813f710 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 10 monclient(hunting): dump: 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:25.953 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:25.954 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:25.954 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:25.954 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:25.954 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:25.954 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:25.954 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:25.954 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:46:25.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662057645 0 0) 0x7f096813b720 con 0x7f09681430c0 2024-03-25T18:46:25.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 10 monclient(hunting): _negotiate 2024-03-25T18:46:25.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:25.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:25.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:46:25.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.953+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f094c004eb0 con 0x7f09681430c0 2024-03-25T18:46:25.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3637300780 0 0) 0x7f0968139d20 con 0x7f096813f710 2024-03-25T18:46:25.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 >> v1:172.21.15.32:6790/0 conn(0x7f09680a98f0 legacy=0x7f096813bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:25.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 >> v1:172.21.15.32:6789/0 conn(0x7f09681430c0 legacy=0x7f09681e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:25.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 10 monclient: _finish_hunting 0 2024-03-25T18:46:25.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 1 monclient: found mon.b 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f096813c250 con 0x7f096813f710 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 10 monclient: _finish_auth 0 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 10 monclient: _check_auth_tickets 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f09657fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 10 monclient: _renew_subs 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09680b71f0 con 0x7f096813f710 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 10 monclient: _renew_subs 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f096812f910 con 0x7f096813f710 2024-03-25T18:46:25.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.967+0000 7f096f21e640 1 librados: init done 2024-03-25T18:46:25.969 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.968+0000 7f096f21e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:25.969 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.968+0000 7f096f21e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:25.969 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.968+0000 7f096f21e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:25.969 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:25.968+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0968139eb0 con 0x7f096813f710 2024-03-25T18:46:26.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.243+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0954004a90 con 0x7f096813f710 2024-03-25T18:46:26.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.243+0000 7f09657fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:26.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.243+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0954004ee0 con 0x7f096813f710 2024-03-25T18:46:26.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.243+0000 7f09657fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.243+0000 7f09657fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.243+0000 7f09657fa640 10 monclient: dump: 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:26.244 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.243+0000 7f0966ffd640 4 set_mon_vals no callback set 2024-03-25T18:46:26.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.243+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f0954005100 con 0x7f096813f710 2024-03-25T18:46:26.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.244+0000 7f096cfba640 10 monclient: get_auth_request con 0x7f094c06dd90 auth_method 0 2024-03-25T18:46:26.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.244+0000 7f09657fa640 10 client.4537.objecter ms_handle_connect 0x7f094c06dd90 2024-03-25T18:46:26.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.250+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f0954054940 con 0x7f096813f710 2024-03-25T18:46:26.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.250+0000 7f09657fa640 10 client.4537.objecter ms_dispatch 0x7f0968000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:46:26.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.250+0000 7f09657fa640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:46:26.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.250+0000 7f09657fa640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:46:26.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.250+0000 7f09657fa640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-25T18:46:26.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.250+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0954050450 con 0x7f096813f710 2024-03-25T18:46:26.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.250+0000 7f09657fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:26.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.250+0000 7f09657fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:46:26.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.483+0000 7f096f21e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:26.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.483+0000 7f096f21e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:26.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.483+0000 7f096f21e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:26.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.483+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f096813c3e0 con 0x7f096813f710 2024-03-25T18:46:26.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.509+0000 7f09657fa640 1 -- 172.21.15.32:0/335019718 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f0954052380 con 0x7f096813f710 2024-03-25T18:46:26.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.509+0000 7f09657fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:26.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.509+0000 7f09657fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:46:26.509 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:46:26.510 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 10 librados: watch_flush enter 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 10 librados: watch_flush exit 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 10 client.4537.objecter successfully canceled tick 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 >> v1:172.21.15.136:6800/1141502808 conn(0x7f094c06dd90 legacy=0x7f094c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 10 monclient: shutdown 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 >> v1:172.21.15.136:6789/0 conn(0x7f096813f710 legacy=0x7f09681df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 shutdown_connections 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 >> 172.21.15.32:0/335019718 conn(0x7f09680b7ea0 msgr2=0x7f0968074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 shutdown_connections 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 1 -- 172.21.15.32:0/335019718 wait complete. 2024-03-25T18:46:26.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:26.511+0000 7f096f21e640 1 librados: shutdown 2024-03-25T18:46:26.522 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:46:28.340 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:40.668 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:28.347 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:40.675 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:28.396 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:40.712 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:28.399 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:07:40.727 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:28.399 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:40.727 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:28.403 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:40.720 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:28.413 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:40.741 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:28.426 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:07:40.743 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:28.441 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:07:40.769 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:33.329 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:46:33.430 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.430+0000 7fd296bed640 10 monclient: get_monmap_and_config 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.430+0000 7fd296bed640 10 monclient: build_initial_monmap 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.430+0000 7fd296bed640 10 monclient: monmap: 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:46:33.430734+0000 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:46:33.430734+0000 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:46:33.431 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:33.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 1 Processor -- start 2024-03-25T18:46:33.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 1 -- start start 2024-03-25T18:46:33.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 10 monclient: init 2024-03-25T18:46:33.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:33.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:46:33.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 10 monclient(hunting): picked mon.noname-b con 0x7fd2900a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:33.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 10 monclient(hunting): picked mon.noname-a con 0x7fd29013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:33.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 10 monclient(hunting): picked mon.noname-c con 0x7fd2901430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:33.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd290142520 con 0x7fd29013f710 2024-03-25T18:46:33.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2900b28a0 con 0x7fd2901430c0 2024-03-25T18:46:33.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2900b2a30 con 0x7fd2900a98f0 2024-03-25T18:46:33.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:33.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.431+0000 7fd296bed640 10 monclient(hunting): authenticate will time out at 6499.620315s 2024-03-25T18:46:33.447 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.447+0000 7fd29518a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd2901430c0 0x7fd29014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43554/0 (socket says 172.21.15.32:43554) 2024-03-25T18:46:33.448 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.447+0000 7fd29518a640 1 -- 172.21.15.32:0/1489178296 learned_addr learned my addr 172.21.15.32:0/1489178296 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:33.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.455+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2949926078 0 0) 0x7fd2900b28a0 con 0x7fd2901430c0 2024-03-25T18:46:33.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.455+0000 7fd28f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:46:33.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.455+0000 7fd28f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:33.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.455+0000 7fd28f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:33.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.455+0000 7fd28f7fe640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:46:33.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.455+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd280003520 con 0x7fd2901430c0 2024-03-25T18:46:33.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1991999976 0 0) 0x7fd280003520 con 0x7fd2901430c0 2024-03-25T18:46:33.466 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 >> v1:172.21.15.32:6790/0 conn(0x7fd2900a98f0 legacy=0x7fd2900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 >> v1:172.21.15.32:6789/0 conn(0x7fd29013f710 legacy=0x7fd2901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2900b44c0 con 0x7fd2901430c0 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd288001a10 con 0x7fd2901430c0 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd296bed640 5 monclient: authenticate success, global_id 4540 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd296bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:33.467 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd28f7fe640 10 monclient: dump: 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:33.468 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:33.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.466+0000 7fd296bed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:33.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd288001be0 con 0x7fd2901430c0 2024-03-25T18:46:33.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd28f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:33.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd28f7fe640 1 -- 172.21.15.32:0/1489178296 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd288002030 con 0x7fd2901430c0 2024-03-25T18:46:33.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd296bed640 10 monclient: get_monmap_and_config success 2024-03-25T18:46:33.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd296bed640 4 set_mon_vals no callback set 2024-03-25T18:46:33.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd28f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd28f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd28f7fe640 10 monclient: dump: 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:33.483 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd296bed640 10 monclient: shutdown 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd296bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd296bed640 1 -- 172.21.15.32:0/1489178296 >> v1:172.21.15.136:6789/0 conn(0x7fd2901430c0 legacy=0x7fd29014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd296bed640 1 -- 172.21.15.32:0/1489178296 shutdown_connections 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd296bed640 1 -- 172.21.15.32:0/1489178296 >> 172.21.15.32:0/1489178296 conn(0x7fd2900b7ea0 msgr2=0x7fd2900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.482+0000 7fd296bed640 1 -- 172.21.15.32:0/1489178296 shutdown_connections 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.483+0000 7fd296bed640 1 -- 172.21.15.32:0/1489178296 wait complete. 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.483+0000 7fd296bed640 10 monclient: build_initial_monmap 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.483+0000 7fd296bed640 10 monclient: monmap: 2024-03-25T18:46:33.486 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 1 librados: starting msgr at 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 1 librados: starting objecter 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 1 Processor -- start 2024-03-25T18:46:33.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 1 -- start start 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 1 librados: setting wanted keys 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 1 librados: calling monclient init 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 10 monclient: init 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.484+0000 7fd296bed640 10 monclient(hunting): picked mon.noname-b con 0x7fd2900a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.485+0000 7fd296bed640 10 monclient(hunting): picked mon.noname-a con 0x7fd29013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.485+0000 7fd296bed640 10 monclient(hunting): picked mon.noname-c con 0x7fd2901430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.485+0000 7fd296bed640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2901db410 con 0x7fd29013f710 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.485+0000 7fd296bed640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2900ab950 con 0x7fd2900a98f0 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.485+0000 7fd296bed640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2900b0df0 con 0x7fd2901430c0 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.485+0000 7fd296bed640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:33.488 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.485+0000 7fd296bed640 10 monclient(hunting): authenticate will time out at 6499.673679s 2024-03-25T18:46:33.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.499+0000 7fd294989640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd2900a98f0 0x7fd2901dad00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43574/0 (socket says 172.21.15.32:43574) 2024-03-25T18:46:33.499 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.499+0000 7fd294989640 1 -- 172.21.15.32:0/3958396517 learned_addr learned my addr 172.21.15.32:0/3958396517 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:33.504 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.504+0000 7fd28d7fa640 10 client.?.objecter ms_handle_connect 0x7fd2900a98f0 2024-03-25T18:46:33.504 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.504+0000 7fd28d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:33.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.510+0000 7fd28d7fa640 10 client.?.objecter ms_handle_connect 0x7fd29013f710 2024-03-25T18:46:33.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.510+0000 7fd28d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:33.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.513+0000 7fd28d7fa640 10 client.?.objecter ms_handle_connect 0x7fd2901430c0 2024-03-25T18:46:33.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.513+0000 7fd28d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:33.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.521+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1360011172 0 0) 0x7fd2901db410 con 0x7fd29013f710 2024-03-25T18:46:33.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.521+0000 7fd28d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:46:33.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.521+0000 7fd28d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:33.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.521+0000 7fd28d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:33.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.521+0000 7fd28d7fa640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:46:33.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.521+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd274004eb0 con 0x7fd29013f710 2024-03-25T18:46:33.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.523+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3665563828 0 0) 0x7fd2900b0df0 con 0x7fd2901430c0 2024-03-25T18:46:33.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.523+0000 7fd28d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:46:33.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.523+0000 7fd28d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:33.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.523+0000 7fd28d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:33.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.523+0000 7fd28d7fa640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:46:33.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.523+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2901db410 con 0x7fd2901430c0 2024-03-25T18:46:33.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.524+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476030598 0 0) 0x7fd2900ab950 con 0x7fd2900a98f0 2024-03-25T18:46:33.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.524+0000 7fd28d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:46:33.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.524+0000 7fd28d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:33.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.524+0000 7fd28d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:33.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.524+0000 7fd28d7fa640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:46:33.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.524+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2900b0df0 con 0x7fd2900a98f0 2024-03-25T18:46:33.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.537+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1977773940 0 0) 0x7fd2901db410 con 0x7fd2901430c0 2024-03-25T18:46:33.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.537+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2900ab950 con 0x7fd2901430c0 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.537+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd288001ef0 con 0x7fd2901430c0 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.537+0000 7fd28d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.537+0000 7fd28d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.537+0000 7fd28d7fa640 10 monclient(hunting): dump: 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.537+0000 7fd28d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:46:33.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.538+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 317036757 0 0) 0x7fd274004eb0 con 0x7fd29013f710 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.538+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd2901db410 con 0x7fd29013f710 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.538+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd27c0018b0 con 0x7fd29013f710 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.538+0000 7fd28d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.538+0000 7fd28d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.538+0000 7fd28d7fa640 10 monclient(hunting): dump: 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:33.540 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:33.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3592939546 0 0) 0x7fd2900ab950 con 0x7fd2901430c0 2024-03-25T18:46:33.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 >> v1:172.21.15.136:6789/0 conn(0x7fd2900a98f0 legacy=0x7fd2901dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 >> v1:172.21.15.32:6789/0 conn(0x7fd29013f710 legacy=0x7fd2900b00e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 1 monclient: found mon.c 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2900b0f80 con 0x7fd2901430c0 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd28d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 10 monclient: _renew_subs 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2900b07f0 con 0x7fd2901430c0 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 10 client.4547.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:46:33.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 10 monclient: _renew_subs 2024-03-25T18:46:33.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:33.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2901d48f0 con 0x7fd2901430c0 2024-03-25T18:46:33.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.546+0000 7fd296bed640 1 librados: init done 2024-03-25T18:46:33.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.547+0000 7fd296bed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:33.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.547+0000 7fd296bed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:33.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.547+0000 7fd296bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:33.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.547+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2901db9b0 con 0x7fd2901430c0 2024-03-25T18:46:33.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.868+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd288002050 con 0x7fd2901430c0 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.868+0000 7fd28d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.868+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd288004fb0 con 0x7fd2901430c0 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.868+0000 7fd28d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.868+0000 7fd28d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.868+0000 7fd28d7fa640 10 monclient: dump: 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:33.869 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:33.870 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:33.870 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:33.870 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:33.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.868+0000 7fd28effd640 4 set_mon_vals no callback set 2024-03-25T18:46:33.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.869+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fd28801af80 con 0x7fd2901430c0 2024-03-25T18:46:33.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.869+0000 7fd294989640 10 monclient: get_auth_request con 0x7fd27406dcd0 auth_method 0 2024-03-25T18:46:33.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.870+0000 7fd28d7fa640 10 client.4547.objecter ms_handle_connect 0x7fd27406dcd0 2024-03-25T18:46:33.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.875+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fd288054ae0 con 0x7fd2901430c0 2024-03-25T18:46:33.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.875+0000 7fd28d7fa640 10 client.4547.objecter ms_dispatch 0x7fd290000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:46:33.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.875+0000 7fd28d7fa640 3 client.4547.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:46:33.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.875+0000 7fd28d7fa640 3 client.4547.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:46:33.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.875+0000 7fd28d7fa640 20 client.4547.objecter dump_active .. 0 homeless 2024-03-25T18:46:33.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.875+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd2880505f0 con 0x7fd2901430c0 2024-03-25T18:46:33.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.875+0000 7fd28d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:33.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:33.875+0000 7fd28d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:46:34.137 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.137+0000 7fd296bed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:34.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.137+0000 7fd296bed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:34.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.137+0000 7fd296bed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.32:6790/0 2024-03-25T18:46:34.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.137+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 --> v1:172.21.15.32:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd2900b1190 con 0x7fd2901430c0 2024-03-25T18:46:34.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.156+0000 7fd28d7fa640 1 -- 172.21.15.32:0/3958396517 <== mon.2 v1:172.21.15.32:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fd288052520 con 0x7fd2901430c0 2024-03-25T18:46:34.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.157+0000 7fd28d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:34.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.157+0000 7fd28d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:46:34.157 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:46:34.157 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:46:34.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 10 librados: watch_flush enter 2024-03-25T18:46:34.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 10 librados: watch_flush exit 2024-03-25T18:46:34.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 20 client.4547.objecter shutdown clearing up homeless session... 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 10 client.4547.objecter successfully canceled tick 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 >> v1:172.21.15.136:6800/1141502808 conn(0x7fd27406dcd0 legacy=0x7fd274070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 10 monclient: shutdown 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 >> v1:172.21.15.32:6790/0 conn(0x7fd2901430c0 legacy=0x7fd2901d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 shutdown_connections 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 >> 172.21.15.32:0/3958396517 conn(0x7fd2900b7ea0 msgr2=0x7fd290074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 shutdown_connections 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 1 -- 172.21.15.32:0/3958396517 wait complete. 2024-03-25T18:46:34.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:34.159+0000 7fd296bed640 1 librados: shutdown 2024-03-25T18:46:34.171 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:46:40.978 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:46:41.099 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.098+0000 7fa2e4b32640 10 monclient: get_monmap_and_config 2024-03-25T18:46:41.099 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.098+0000 7fa2e4b32640 10 monclient: build_initial_monmap 2024-03-25T18:46:41.099 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.098+0000 7fa2e4b32640 10 monclient: monmap: 2024-03-25T18:46:41.099 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:41.099 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:41.100 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:46:41.099698+0000 2024-03-25T18:46:41.100 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:46:41.099698+0000 2024-03-25T18:46:41.100 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:41.100 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:41.100 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:41.100 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:46:41.100 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:46:41.100 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:41.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.099+0000 7fa2e4b32640 1 Processor -- start 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.099+0000 7fa2e4b32640 1 -- start start 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.099+0000 7fa2e4b32640 10 monclient: init 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.099+0000 7fa2e4b32640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.099+0000 7fa2e4b32640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.099+0000 7fa2e4b32640 10 monclient(hunting): picked mon.noname-c con 0x7fa2e00a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.100+0000 7fa2e4b32640 10 monclient(hunting): picked mon.noname-b con 0x7fa2e013f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.100+0000 7fa2e4b32640 10 monclient(hunting): picked mon.noname-a con 0x7fa2e01430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.100+0000 7fa2e4b32640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e0142520 con 0x7fa2e01430c0 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.100+0000 7fa2e4b32640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e00b28a0 con 0x7fa2e00a98f0 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.100+0000 7fa2e4b32640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e00b2a30 con 0x7fa2e013f710 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.100+0000 7fa2e4b32640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:41.102 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.100+0000 7fa2e4b32640 10 monclient(hunting): authenticate will time out at 6507.289607s 2024-03-25T18:46:41.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.109+0000 7fa2d7fff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fa2e013f710 0x7fa2e01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:39874/0 (socket says 172.21.15.32:39874) 2024-03-25T18:46:41.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.109+0000 7fa2d7fff640 1 -- 172.21.15.32:0/1493600872 learned_addr learned my addr 172.21.15.32:0/1493600872 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:41.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.121+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297377710 0 0) 0x7fa2e00b2a30 con 0x7fa2e013f710 2024-03-25T18:46:41.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.121+0000 7fa2ddd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:46:41.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.121+0000 7fa2ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:41.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.121+0000 7fa2ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:41.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.121+0000 7fa2ddd9b640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:46:41.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.121+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2cc003520 con 0x7fa2e013f710 2024-03-25T18:46:41.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.122+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2882430360 0 0) 0x7fa2e00b28a0 con 0x7fa2e00a98f0 2024-03-25T18:46:41.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.122+0000 7fa2ddd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:46:41.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.122+0000 7fa2ddd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:41.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.122+0000 7fa2ddd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:41.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.122+0000 7fa2ddd9b640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:46:41.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.122+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e00b2a30 con 0x7fa2e00a98f0 2024-03-25T18:46:41.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3202803922 0 0) 0x7fa2e00b2a30 con 0x7fa2e00a98f0 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 >> v1:172.21.15.32:6790/0 conn(0x7fa2e013f710 legacy=0x7fa2e01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 >> v1:172.21.15.32:6789/0 conn(0x7fa2e01430c0 legacy=0x7fa2e014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 1 monclient: found mon.noname-c 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e00b44c0 con 0x7fa2e00a98f0 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 10 monclient: _finish_auth 0 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2ddd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2e4b32640 5 monclient: authenticate success, global_id 4546 2024-03-25T18:46:41.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.127+0000 7fa2e4b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:41.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.138+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa2d0001630 con 0x7fa2e00a98f0 2024-03-25T18:46:41.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.138+0000 7fa2ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:41.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.138+0000 7fa2ddd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:46:41.139 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.138+0000 7fa2ddd9b640 10 monclient: dump: 2024-03-25T18:46:41.139 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:41.139 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:41.139 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.140 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.140 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:41.140 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:41.140 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:41.140 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:41.140 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:41.140 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:41.140 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.138+0000 7fa2e4b32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.162+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2d00018c0 con 0x7fa2e00a98f0 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.162+0000 7fa2ddd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.162+0000 7fa2ddd9b640 1 -- 172.21.15.32:0/1493600872 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa2d0001bd0 con 0x7fa2e00a98f0 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.162+0000 7fa2ddd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.162+0000 7fa2ddd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.162+0000 7fa2ddd9b640 10 monclient: dump: 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:41.164 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:41.165 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:41.165 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:41.165 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:41.165 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.162+0000 7fa2e4b32640 10 monclient: get_monmap_and_config success 2024-03-25T18:46:41.165 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.162+0000 7fa2e4b32640 4 set_mon_vals no callback set 2024-03-25T18:46:41.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.163+0000 7fa2e4b32640 10 monclient: shutdown 2024-03-25T18:46:41.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.163+0000 7fa2e4b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.163+0000 7fa2e4b32640 1 -- 172.21.15.32:0/1493600872 >> v1:172.21.15.136:6789/0 conn(0x7fa2e00a98f0 legacy=0x7fa2e00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.163+0000 7fa2e4b32640 1 -- 172.21.15.32:0/1493600872 shutdown_connections 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.163+0000 7fa2e4b32640 1 -- 172.21.15.32:0/1493600872 >> 172.21.15.32:0/1493600872 conn(0x7fa2e00b7ea0 msgr2=0x7fa2e00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.163+0000 7fa2e4b32640 1 -- 172.21.15.32:0/1493600872 shutdown_connections 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.163+0000 7fa2e4b32640 1 -- 172.21.15.32:0/1493600872 wait complete. 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.164+0000 7fa2e4b32640 10 monclient: build_initial_monmap 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.164+0000 7fa2e4b32640 10 monclient: monmap: 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:46:41.168 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.165+0000 7fa2e4b32640 1 librados: starting msgr at 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.165+0000 7fa2e4b32640 1 librados: starting objecter 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.165+0000 7fa2e4b32640 1 Processor -- start 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.165+0000 7fa2e4b32640 1 -- start start 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.165+0000 7fa2e4b32640 1 librados: setting wanted keys 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.165+0000 7fa2e4b32640 1 librados: calling monclient init 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.165+0000 7fa2e4b32640 10 monclient: init 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 10 monclient(hunting): picked mon.noname-c con 0x7fa2e00a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 10 monclient(hunting): picked mon.noname-b con 0x7fa2e013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:41.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 10 monclient(hunting): picked mon.noname-a con 0x7fa2e01430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:41.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e01db790 con 0x7fa2e01430c0 2024-03-25T18:46:41.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e014e8d0 con 0x7fa2e013f710 2024-03-25T18:46:41.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2e01da880 con 0x7fa2e00a98f0 2024-03-25T18:46:41.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:41.170 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.166+0000 7fa2e4b32640 10 monclient(hunting): authenticate will time out at 6507.355688s 2024-03-25T18:46:41.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.175+0000 7fa2ded9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fa2e01430c0 0x7fa2e01e1bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:47372/0 (socket says 172.21.15.32:47372) 2024-03-25T18:46:41.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.175+0000 7fa2ded9d640 1 -- 172.21.15.32:0/3403140215 learned_addr learned my addr 172.21.15.32:0/3403140215 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:41.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.176+0000 7fa2d6ffd640 10 client.?.objecter ms_handle_connect 0x7fa2e01430c0 2024-03-25T18:46:41.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.176+0000 7fa2d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:41.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.178+0000 7fa2d6ffd640 10 client.?.objecter ms_handle_connect 0x7fa2e013f710 2024-03-25T18:46:41.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.178+0000 7fa2d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:41.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.181+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4198509876 0 0) 0x7fa2e01db790 con 0x7fa2e01430c0 2024-03-25T18:46:41.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.181+0000 7fa2d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:41.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.181+0000 7fa2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:41.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.181+0000 7fa2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:41.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.182+0000 7fa2d6ffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:46:41.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.182+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2c0004eb0 con 0x7fa2e01430c0 2024-03-25T18:46:41.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.183+0000 7fa2d6ffd640 10 client.?.objecter ms_handle_connect 0x7fa2e00a98f0 2024-03-25T18:46:41.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.183+0000 7fa2d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:41.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.185+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1934703713 0 0) 0x7fa2e014e8d0 con 0x7fa2e013f710 2024-03-25T18:46:41.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.185+0000 7fa2d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:41.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.185+0000 7fa2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:41.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.185+0000 7fa2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:41.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.185+0000 7fa2d6ffd640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:46:41.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.185+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e01db790 con 0x7fa2e013f710 2024-03-25T18:46:41.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.196+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2462087202 0 0) 0x7fa2e01db790 con 0x7fa2e013f710 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.196+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e014e8d0 con 0x7fa2e013f710 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.196+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa2c8001910 con 0x7fa2e013f710 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.196+0000 7fa2d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.196+0000 7fa2d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.196+0000 7fa2d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:41.198 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:41.199 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:41.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.196+0000 7fa2d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:46:41.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.198+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453159973 0 0) 0x7fa2e01da880 con 0x7fa2e00a98f0 2024-03-25T18:46:41.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.198+0000 7fa2d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:41.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.198+0000 7fa2d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:41.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.198+0000 7fa2d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:41.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.198+0000 7fa2d6ffd640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:46:41.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.198+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2e01db790 con 0x7fa2e00a98f0 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.205+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4243259201 0 0) 0x7fa2c0004eb0 con 0x7fa2e01430c0 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.205+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2e01da880 con 0x7fa2e01430c0 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.205+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa2d8001c30 con 0x7fa2e01430c0 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.205+0000 7fa2d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.205+0000 7fa2d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.205+0000 7fa2d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.207 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.208 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:41.208 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:41.208 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:41.208 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:41.208 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:41.208 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:41.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1531808667 0 0) 0x7fa2e014e8d0 con 0x7fa2e013f710 2024-03-25T18:46:41.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 >> v1:172.21.15.32:6790/0 conn(0x7fa2e00a98f0 legacy=0x7fa2e01db080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:41.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 >> v1:172.21.15.32:6789/0 conn(0x7fa2e01430c0 legacy=0x7fa2e01e1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:41.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:46:41.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 1 monclient: found mon.b 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e01dab60 con 0x7fa2e013f710 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2e4b32640 5 monclient: authenticate success, global_id 4549 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2e4b32640 10 monclient: _renew_subs 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2e4b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.206+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e00b1460 con 0x7fa2e013f710 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.207+0000 7fa2e4b32640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.207+0000 7fa2e4b32640 10 monclient: _renew_subs 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.207+0000 7fa2e4b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.207+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e014ddc0 con 0x7fa2e013f710 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.207+0000 7fa2e4b32640 1 librados: init done 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.208+0000 7fa2e4b32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:41.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.208+0000 7fa2e4b32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:41.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.208+0000 7fa2e4b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:41.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.208+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e00627b0 con 0x7fa2e013f710 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.477+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2c8001d20 con 0x7fa2e013f710 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.477+0000 7fa2d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.477+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fa2c8004980 con 0x7fa2e013f710 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.477+0000 7fa2d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.477+0000 7fa2d6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.477+0000 7fa2d6ffd640 10 monclient: dump: 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:41.479 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:41.480 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:41.480 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:41.480 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:41.480 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:41.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.477+0000 7fa2dd59a640 4 set_mon_vals no callback set 2024-03-25T18:46:41.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.478+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fa2c8004ba0 con 0x7fa2e013f710 2024-03-25T18:46:41.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.478+0000 7fa2de59c640 10 monclient: get_auth_request con 0x7fa2c006dfb0 auth_method 0 2024-03-25T18:46:41.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.479+0000 7fa2d6ffd640 10 client.4549.objecter ms_handle_connect 0x7fa2c006dfb0 2024-03-25T18:46:41.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.483+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fa2c8054670 con 0x7fa2e013f710 2024-03-25T18:46:41.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.483+0000 7fa2d6ffd640 10 client.4549.objecter ms_dispatch 0x7fa2e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:46:41.484 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.483+0000 7fa2d6ffd640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:46:41.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.483+0000 7fa2d6ffd640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:46:41.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.483+0000 7fa2d6ffd640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-25T18:46:41.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.483+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2c80b18b0 con 0x7fa2e013f710 2024-03-25T18:46:41.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.483+0000 7fa2d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:41.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.483+0000 7fa2d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:46:41.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.733+0000 7fa2e4b32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:41.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.733+0000 7fa2e4b32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:41.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.733+0000 7fa2e4b32640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:41.735 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.733+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2e0074cc0 con 0x7fa2e013f710 2024-03-25T18:46:41.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.755+0000 7fa2d6ffd640 1 -- 172.21.15.32:0/3403140215 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fa2c804ffb0 con 0x7fa2e013f710 2024-03-25T18:46:41.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.756+0000 7fa2d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:41.757 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.756+0000 7fa2d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:46:41.757 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:46:41.757 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:46:41.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.757+0000 7fa2e4b32640 10 librados: watch_flush enter 2024-03-25T18:46:41.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 10 librados: watch_flush exit 2024-03-25T18:46:41.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-25T18:46:41.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 10 client.4549.objecter successfully canceled tick 2024-03-25T18:46:41.759 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 >> v1:172.21.15.136:6800/1141502808 conn(0x7fa2c006dfb0 legacy=0x7fa2c0070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:41.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 10 monclient: shutdown 2024-03-25T18:46:41.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:41.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 >> v1:172.21.15.136:6789/0 conn(0x7fa2e013f710 legacy=0x7fa2e00b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:41.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 shutdown_connections 2024-03-25T18:46:41.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 >> 172.21.15.32:0/3403140215 conn(0x7fa2e00b7ea0 msgr2=0x7fa2e0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:41.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 shutdown_connections 2024-03-25T18:46:41.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 1 -- 172.21.15.32:0/3403140215 wait complete. 2024-03-25T18:46:41.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:41.758+0000 7fa2e4b32640 1 librados: shutdown 2024-03-25T18:46:41.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:46:48.318 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:00.646 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.371 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:00.699 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.404 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:00.732 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.416 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:00.733 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.422 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:00.738 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.443 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:00.760 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.448 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:00.775 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.459 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:00.787 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.484 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:00.811 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:46:48.578 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:46:48.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.674+0000 7f8abeac8640 10 monclient: get_monmap_and_config 2024-03-25T18:46:48.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.674+0000 7f8abeac8640 10 monclient: build_initial_monmap 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.674+0000 7f8abeac8640 10 monclient: monmap: 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:46:48.676043+0000 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:46:48.676043+0000 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:46:48.676 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.675+0000 7f8abeac8640 1 Processor -- start 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.675+0000 7f8abeac8640 1 -- start start 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.675+0000 7f8abeac8640 10 monclient: init 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab80a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab81430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab8142520 con 0x7f8ab813f710 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80b28a0 con 0x7f8ab81430c0 2024-03-25T18:46:48.678 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80b2a30 con 0x7f8ab80a98f0 2024-03-25T18:46:48.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:48.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abeac8640 10 monclient(hunting): authenticate will time out at 6514.865835s 2024-03-25T18:46:48.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abd065640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8ab81430c0 0x7f8ab814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:59972/0 (socket says 172.21.15.32:59972) 2024-03-25T18:46:48.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.676+0000 7f8abd065640 1 -- 172.21.15.32:0/769676405 learned_addr learned my addr 172.21.15.32:0/769676405 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:48.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.719+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676396859 0 0) 0x7f8ab8142520 con 0x7f8ab813f710 2024-03-25T18:46:48.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.719+0000 7f8ab77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:46:48.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.719+0000 7f8ab77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:48.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.719+0000 7f8ab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:48.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.719+0000 7f8ab77fe640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:46:48.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.719+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8aa8003520 con 0x7f8ab813f710 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4126532370 0 0) 0x7f8aa8003520 con 0x7f8ab813f710 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 >> v1:172.21.15.32:6790/0 conn(0x7f8ab80a98f0 legacy=0x7f8ab80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 >> v1:172.21.15.136:6789/0 conn(0x7f8ab81430c0 legacy=0x7f8ab814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 1 monclient: found mon.noname-a 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab80b44c0 con 0x7f8ab813f710 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 10 monclient: _finish_auth 0 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8abeac8640 5 monclient: authenticate success, global_id 4572 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8aa4001690 con 0x7f8ab813f710 2024-03-25T18:46:48.786 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8ab77fe640 10 monclient: dump: 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:48.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.784+0000 7f8abeac8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8aa40017f0 con 0x7f8ab813f710 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8ab77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8ab77fe640 1 -- 172.21.15.32:0/769676405 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8aa4001c00 con 0x7f8ab813f710 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8abeac8640 10 monclient: get_monmap_and_config success 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8abeac8640 4 set_mon_vals no callback set 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8ab77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8ab77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8ab77fe640 10 monclient: dump: 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:48.792 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:48.793 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:48.793 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:48.793 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:48.793 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:48.793 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:48.793 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8abeac8640 10 monclient: shutdown 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8abeac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8abeac8640 1 -- 172.21.15.32:0/769676405 >> v1:172.21.15.32:6789/0 conn(0x7f8ab813f710 legacy=0x7f8ab81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8abeac8640 1 -- 172.21.15.32:0/769676405 shutdown_connections 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8abeac8640 1 -- 172.21.15.32:0/769676405 >> 172.21.15.32:0/769676405 conn(0x7f8ab80b7ea0 msgr2=0x7f8ab80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.791+0000 7f8abeac8640 1 -- 172.21.15.32:0/769676405 shutdown_connections 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.792+0000 7f8abeac8640 1 -- 172.21.15.32:0/769676405 wait complete. 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.792+0000 7f8abeac8640 10 monclient: build_initial_monmap 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.792+0000 7f8abeac8640 10 monclient: monmap: 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:46:48.795 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.792+0000 7f8abeac8640 1 librados: starting msgr at 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.792+0000 7f8abeac8640 1 librados: starting objecter 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 1 Processor -- start 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 1 -- start start 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 1 librados: setting wanted keys 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 1 librados: calling monclient init 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 10 monclient: init 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:48.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:46:48.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 10 monclient(hunting): picked mon.noname-a con 0x7f8ab80a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:48.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 10 monclient(hunting): picked mon.noname-b con 0x7f8ab813f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:48.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 10 monclient(hunting): picked mon.noname-c con 0x7f8ab81430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:48.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab8138980 con 0x7f8ab80a98f0 2024-03-25T18:46:48.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab8138640 con 0x7f8ab813f710 2024-03-25T18:46:48.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ab80b1700 con 0x7f8ab81430c0 2024-03-25T18:46:48.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:48.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.793+0000 7f8abeac8640 10 monclient(hunting): authenticate will time out at 6514.983353s 2024-03-25T18:46:48.804 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.803+0000 7f8ab7fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f8ab813f710 0x7f8ab81da8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:60022/0 (socket says 172.21.15.32:60022) 2024-03-25T18:46:48.804 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.803+0000 7f8ab7fff640 1 -- 172.21.15.32:0/3166956938 learned_addr learned my addr 172.21.15.32:0/3166956938 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:48.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.804+0000 7f8ab57fa640 10 client.?.objecter ms_handle_connect 0x7f8ab813f710 2024-03-25T18:46:48.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.804+0000 7f8ab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:48.813 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.812+0000 7f8ab57fa640 10 client.?.objecter ms_handle_connect 0x7f8ab80a98f0 2024-03-25T18:46:48.813 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.812+0000 7f8ab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:48.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328388749 0 0) 0x7f8ab8138640 con 0x7f8ab813f710 2024-03-25T18:46:48.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:46:48.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:48.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:48.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 10 monclient(hunting): my global_id is 4552 2024-03-25T18:46:48.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a9c004eb0 con 0x7f8ab813f710 2024-03-25T18:46:48.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357629152 0 0) 0x7f8ab8138980 con 0x7f8ab80a98f0 2024-03-25T18:46:48.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:46:48.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:48.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:48.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:46:48.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.828+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ab8138640 con 0x7f8ab80a98f0 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.834+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2300166584 0 0) 0x7f8a9c004eb0 con 0x7f8ab813f710 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.834+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ab8138980 con 0x7f8ab813f710 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.834+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8aa4001690 con 0x7f8ab813f710 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.834+0000 7f8ab57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.834+0000 7f8ab57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.834+0000 7f8ab57fa640 10 monclient(hunting): dump: 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:48.836 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:48.837 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:48.837 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:48.837 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:48.837 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:48.837 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.834+0000 7f8ab57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:46:48.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.849+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3874372634 0 0) 0x7f8ab8138640 con 0x7f8ab80a98f0 2024-03-25T18:46:48.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.849+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a9c004eb0 con 0x7f8ab80a98f0 2024-03-25T18:46:48.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.849+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8aac0018a0 con 0x7f8ab80a98f0 2024-03-25T18:46:48.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.849+0000 7f8ab57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:48.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.849+0000 7f8ab57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:46:48.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.849+0000 7f8ab57fa640 10 monclient(hunting): dump: 2024-03-25T18:46:48.850 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:48.851 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:48.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.854+0000 7f8ab57fa640 10 client.?.objecter ms_handle_connect 0x7f8ab81430c0 2024-03-25T18:46:48.855 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.854+0000 7f8ab57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2194049593 0 0) 0x7f8ab8138980 con 0x7f8ab813f710 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 >> v1:172.21.15.32:6790/0 conn(0x7f8ab81430c0 legacy=0x7f8ab81db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 >> v1:172.21.15.32:6789/0 conn(0x7f8ab80a98f0 legacy=0x7f8ab8137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 1 monclient: found mon.b 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ab80b1290 con 0x7f8ab813f710 2024-03-25T18:46:48.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 10 monclient: _finish_auth 0 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8ab57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 5 monclient: authenticate success, global_id 4552 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 10 monclient: _renew_subs 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ab81e2eb0 con 0x7f8ab813f710 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 10 client.4552.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 10 monclient: _renew_subs 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ab81e3220 con 0x7f8ab813f710 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.855+0000 7f8abeac8640 1 librados: init done 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.856+0000 7f8abeac8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:48.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.856+0000 7f8abeac8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:48.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.856+0000 7f8abeac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:48.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:48.856+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ab80b14e0 con 0x7f8ab813f710 2024-03-25T18:46:49.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.134+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8aa40038a0 con 0x7f8ab813f710 2024-03-25T18:46:49.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.134+0000 7f8ab57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.134+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8aa4003b50 con 0x7f8ab813f710 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.134+0000 7f8ab57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.134+0000 7f8ab57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.134+0000 7f8ab57fa640 10 monclient: dump: 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:49.136 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:49.137 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:49.137 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:49.137 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:49.137 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:49.137 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:49.137 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.134+0000 7f8ab6ffd640 4 set_mon_vals no callback set 2024-03-25T18:46:49.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.135+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f8aa4003d70 con 0x7f8ab813f710 2024-03-25T18:46:49.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.135+0000 7f8abc864640 10 monclient: get_auth_request con 0x7f8a9c06de80 auth_method 0 2024-03-25T18:46:49.138 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.135+0000 7f8ab57fa640 10 client.4552.objecter ms_handle_connect 0x7f8a9c06de80 2024-03-25T18:46:49.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.140+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f8aa4054810 con 0x7f8ab813f710 2024-03-25T18:46:49.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.140+0000 7f8ab57fa640 10 client.4552.objecter ms_dispatch 0x7f8ab8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:46:49.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.140+0000 7f8ab57fa640 3 client.4552.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:46:49.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.140+0000 7f8ab57fa640 3 client.4552.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:46:49.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.140+0000 7f8ab57fa640 20 client.4552.objecter dump_active .. 0 homeless 2024-03-25T18:46:49.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.140+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8aa4050320 con 0x7f8ab813f710 2024-03-25T18:46:49.141 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.140+0000 7f8ab57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:49.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.140+0000 7f8ab57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:46:49.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.389+0000 7f8abeac8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:49.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.389+0000 7f8abeac8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:49.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.389+0000 7f8abeac8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:46:49.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.389+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ab80627b0 con 0x7f8ab813f710 2024-03-25T18:46:49.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.408+0000 7f8ab57fa640 1 -- 172.21.15.32:0/3166956938 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f8aa4052250 con 0x7f8ab813f710 2024-03-25T18:46:49.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.408+0000 7f8ab57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:49.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.408+0000 7f8ab57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:46:49.410 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:46:49.410 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 10 librados: watch_flush enter 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 10 librados: watch_flush exit 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 20 client.4552.objecter shutdown clearing up homeless session... 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 10 client.4552.objecter successfully canceled tick 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 >> v1:172.21.15.136:6800/1141502808 conn(0x7f8a9c06de80 legacy=0x7f8a9c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 10 monclient: shutdown 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 >> v1:172.21.15.136:6789/0 conn(0x7f8ab813f710 legacy=0x7f8ab81da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 shutdown_connections 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 >> 172.21.15.32:0/3166956938 conn(0x7f8ab80b7ea0 msgr2=0x7f8ab8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 shutdown_connections 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 1 -- 172.21.15.32:0/3166956938 wait complete. 2024-03-25T18:46:49.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:49.411+0000 7f8abeac8640 1 librados: shutdown 2024-03-25T18:46:49.424 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:46:56.232 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:46:56.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.330+0000 7fe6ae86e640 10 monclient: get_monmap_and_config 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.330+0000 7fe6ae86e640 10 monclient: build_initial_monmap 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.330+0000 7fe6ae86e640 10 monclient: monmap: 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:46:56.332034+0000 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:46:56.332034+0000 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:46:56.332 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 1 Processor -- start 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 1 -- start start 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 10 monclient: init 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 10 monclient(hunting): picked mon.noname-b con 0x7fe6a80b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 10 monclient(hunting): picked mon.noname-c con 0x7fe6a80b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 10 monclient(hunting): picked mon.noname-a con 0x7fe6a814be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a80af940 con 0x7fe6a814be90 2024-03-25T18:46:56.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a80b0a70 con 0x7fe6a80b1730 2024-03-25T18:46:56.335 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a80b0d50 con 0x7fe6a80b4d60 2024-03-25T18:46:56.335 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:56.335 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.331+0000 7fe6ae86e640 10 monclient(hunting): authenticate will time out at 6522.521511s 2024-03-25T18:46:56.335 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.332+0000 7fe6a7fff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fe6a80b4d60 0x7fe6a813f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:50544/0 (socket says 172.21.15.32:50544) 2024-03-25T18:46:56.335 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.332+0000 7fe6a7fff640 1 -- 172.21.15.32:0/3656247124 learned_addr learned my addr 172.21.15.32:0/3656247124 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:56.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.359+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946008798 0 0) 0x7fe6a80af940 con 0x7fe6a814be90 2024-03-25T18:46:56.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.359+0000 7fe6a6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:46:56.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.359+0000 7fe6a6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:56.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.359+0000 7fe6a6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:56.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.359+0000 7fe6a6ffd640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:46:56.360 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.359+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe690003520 con 0x7fe6a814be90 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4028788981 0 0) 0x7fe690003520 con 0x7fe6a814be90 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 >> v1:172.21.15.32:6790/0 conn(0x7fe6a80b4d60 legacy=0x7fe6a813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 >> v1:172.21.15.136:6789/0 conn(0x7fe6a80b1730 legacy=0x7fe6a80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a80b1030 con 0x7fe6a814be90 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:46:56.380 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe6a0001a10 con 0x7fe6a814be90 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6ae86e640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6ae86e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6a6ffd640 10 monclient: dump: 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:56.381 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:56.382 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:56.382 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:56.382 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:56.382 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.378+0000 7fe6ae86e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:46:56.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6a0001b70 con 0x7fe6a814be90 2024-03-25T18:46:56.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6a6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:56.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6a6ffd640 1 -- 172.21.15.32:0/3656247124 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe6a0001f80 con 0x7fe6a814be90 2024-03-25T18:46:56.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6ae86e640 10 monclient: get_monmap_and_config success 2024-03-25T18:46:56.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6ae86e640 4 set_mon_vals no callback set 2024-03-25T18:46:56.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6a6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:56.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6a6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:46:56.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6a6ffd640 10 monclient: dump: 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6ae86e640 10 monclient: shutdown 2024-03-25T18:46:56.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6ae86e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:56.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.401+0000 7fe6ae86e640 1 -- 172.21.15.32:0/3656247124 >> v1:172.21.15.32:6789/0 conn(0x7fe6a814be90 legacy=0x7fe6a814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:56.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.402+0000 7fe6ae86e640 1 -- 172.21.15.32:0/3656247124 shutdown_connections 2024-03-25T18:46:56.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.402+0000 7fe6ae86e640 1 -- 172.21.15.32:0/3656247124 >> 172.21.15.32:0/3656247124 conn(0x7fe6a80617a0 msgr2=0x7fe6a8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:56.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.402+0000 7fe6ae86e640 1 -- 172.21.15.32:0/3656247124 shutdown_connections 2024-03-25T18:46:56.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.402+0000 7fe6ae86e640 1 -- 172.21.15.32:0/3656247124 wait complete. 2024-03-25T18:46:56.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.402+0000 7fe6ae86e640 10 monclient: build_initial_monmap 2024-03-25T18:46:56.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.402+0000 7fe6ae86e640 10 monclient: monmap: 2024-03-25T18:46:56.409 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:46:56.409 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:46:56.409 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:46:56.409 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 librados: starting msgr at 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 librados: starting objecter 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 Processor -- start 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 -- start start 2024-03-25T18:46:56.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 librados: setting wanted keys 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 librados: calling monclient init 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 10 monclient: init 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 10 monclient: _reopen_session rank -1 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 10 monclient(hunting): picked mon.noname-a con 0x7fe6a80b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 10 monclient(hunting): picked mon.noname-b con 0x7fe6a80b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 10 monclient(hunting): picked mon.noname-c con 0x7fe6a814be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a8140810 con 0x7fe6a80b1730 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a8140380 con 0x7fe6a80b4d60 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6a8140510 con 0x7fe6a814be90 2024-03-25T18:46:56.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 10 monclient(hunting): _renew_subs 2024-03-25T18:46:56.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.403+0000 7fe6ae86e640 10 monclient(hunting): authenticate will time out at 6522.593452s 2024-03-25T18:46:56.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.415+0000 7fe6a7fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe6a80b1730 0x7fe6a8134ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:45650/0 (socket says 172.21.15.32:45650) 2024-03-25T18:46:56.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.415+0000 7fe6a7fff640 1 -- 172.21.15.32:0/1858531884 learned_addr learned my addr 172.21.15.32:0/1858531884 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:46:56.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.421+0000 7fe6a4ff9640 10 client.?.objecter ms_handle_connect 0x7fe6a80b1730 2024-03-25T18:46:56.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.422+0000 7fe6a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:56.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.426+0000 7fe6a4ff9640 10 client.?.objecter ms_handle_connect 0x7fe6a80b4d60 2024-03-25T18:46:56.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.426+0000 7fe6a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:56.432 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.431+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2635624329 0 0) 0x7fe6a8140810 con 0x7fe6a80b1730 2024-03-25T18:46:56.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.431+0000 7fe6a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:46:56.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.431+0000 7fe6a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:56.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.431+0000 7fe6a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:56.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.431+0000 7fe6a4ff9640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:46:56.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.431+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe694004eb0 con 0x7fe6a80b1730 2024-03-25T18:46:56.439 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.438+0000 7fe6a4ff9640 10 client.?.objecter ms_handle_connect 0x7fe6a814be90 2024-03-25T18:46:56.439 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.438+0000 7fe6a4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:46:56.447 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.446+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 315716895 0 0) 0x7fe694004eb0 con 0x7fe6a80b1730 2024-03-25T18:46:56.447 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.446+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe6a8140810 con 0x7fe6a80b1730 2024-03-25T18:46:56.447 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.446+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe6980018b0 con 0x7fe6a80b1730 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.446+0000 7fe6a4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.446+0000 7fe6a4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.446+0000 7fe6a4ff9640 10 monclient(hunting): dump: 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:56.448 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:56.449 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:56.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.446+0000 7fe6a4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:46:56.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.447+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518552090 0 0) 0x7fe6a8140380 con 0x7fe6a80b4d60 2024-03-25T18:46:56.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.447+0000 7fe6a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:46:56.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.447+0000 7fe6a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:56.449 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.447+0000 7fe6a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:56.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.447+0000 7fe6a4ff9640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:46:56.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.447+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe694004eb0 con 0x7fe6a80b4d60 2024-03-25T18:46:56.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.452+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2965458511 0 0) 0x7fe6a8140510 con 0x7fe6a814be90 2024-03-25T18:46:56.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.452+0000 7fe6a4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:46:56.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.452+0000 7fe6a4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:46:56.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.452+0000 7fe6a4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:46:56.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.452+0000 7fe6a4ff9640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:46:56.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.452+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe6a8140380 con 0x7fe6a814be90 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2034200221 0 0) 0x7fe6a8140810 con 0x7fe6a80b1730 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 >> v1:172.21.15.32:6790/0 conn(0x7fe6a814be90 legacy=0x7fe6a81e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 >> v1:172.21.15.136:6789/0 conn(0x7fe6a80b4d60 legacy=0x7fe6a81d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 1 monclient: found mon.a 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6a81355e0 con 0x7fe6a80b1730 2024-03-25T18:46:56.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6a4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 5 monclient: authenticate success, global_id 4581 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 10 monclient: _renew_subs 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6a81357d0 con 0x7fe6a80b1730 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 10 client.4581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 10 monclient: _renew_subs 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6a8144da0 con 0x7fe6a80b1730 2024-03-25T18:46:56.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.457+0000 7fe6ae86e640 1 librados: init done 2024-03-25T18:46:56.461 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.458+0000 7fe6ae86e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:56.461 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.458+0000 7fe6ae86e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:56.461 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.458+0000 7fe6ae86e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:46:56.461 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.458+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6a8141c10 con 0x7fe6a80b1730 2024-03-25T18:46:56.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.796+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe6980047e0 con 0x7fe6a80b1730 2024-03-25T18:46:56.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.796+0000 7fe6a4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:46:56.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.796+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe698004c30 con 0x7fe6a80b1730 2024-03-25T18:46:56.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.796+0000 7fe6a4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:46:56.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.796+0000 7fe6a4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:46:56.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.796+0000 7fe6a4ff9640 10 monclient: dump: 2024-03-25T18:46:56.797 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:46:56.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.796+0000 7fe6a67fc640 4 set_mon_vals no callback set 2024-03-25T18:46:56.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.796+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fe698004e50 con 0x7fe6a80b1730 2024-03-25T18:46:56.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.797+0000 7fe6a77fe640 10 monclient: get_auth_request con 0x7fe69406df30 auth_method 0 2024-03-25T18:46:56.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.797+0000 7fe6a4ff9640 10 client.4581.objecter ms_handle_connect 0x7fe69406df30 2024-03-25T18:46:56.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.803+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fe698054810 con 0x7fe6a80b1730 2024-03-25T18:46:56.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.803+0000 7fe6a4ff9640 10 client.4581.objecter ms_dispatch 0x7fe6a8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:46:56.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.803+0000 7fe6a4ff9640 3 client.4581.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:46:56.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.803+0000 7fe6a4ff9640 3 client.4581.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:46:56.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.804+0000 7fe6a4ff9640 20 client.4581.objecter dump_active .. 0 homeless 2024-03-25T18:46:56.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.804+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe698050320 con 0x7fe6a80b1730 2024-03-25T18:46:56.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.804+0000 7fe6a4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:46:56.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:56.804+0000 7fe6a4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:46:57.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.050+0000 7fe6ae86e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:57.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.050+0000 7fe6ae86e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:57.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.050+0000 7fe6ae86e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:46:57.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.050+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6a80627b0 con 0x7fe6a80b1730 2024-03-25T18:46:57.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.091+0000 7fe6a4ff9640 1 -- 172.21.15.32:0/1858531884 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fe698052250 con 0x7fe6a80b1730 2024-03-25T18:46:57.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.091+0000 7fe6a4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:46:57.092 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.091+0000 7fe6a4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:46:57.093 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:46:57.093 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:46:57.095 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.093+0000 7fe6ae86e640 10 librados: watch_flush enter 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 10 librados: watch_flush exit 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 20 client.4581.objecter shutdown clearing up homeless session... 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 10 client.4581.objecter successfully canceled tick 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 >> v1:172.21.15.136:6800/1141502808 conn(0x7fe69406df30 legacy=0x7fe6940703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 10 monclient: shutdown 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 >> v1:172.21.15.32:6789/0 conn(0x7fe6a80b1730 legacy=0x7fe6a8134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 shutdown_connections 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 >> 172.21.15.32:0/1858531884 conn(0x7fe6a80617a0 msgr2=0x7fe6a80b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 shutdown_connections 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 1 -- 172.21.15.32:0/1858531884 wait complete. 2024-03-25T18:46:57.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:46:57.094+0000 7fe6ae86e640 1 librados: shutdown 2024-03-25T18:46:57.108 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:47:03.914 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.030+0000 7f5f67b9d640 10 monclient: get_monmap_and_config 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.030+0000 7f5f67b9d640 10 monclient: build_initial_monmap 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.030+0000 7f5f67b9d640 10 monclient: monmap: 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:47:04.030504+0000 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:47:04.030504+0000 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:04.030 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:47:04.031 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:47:04.031 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:04.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.030+0000 7f5f67b9d640 1 Processor -- start 2024-03-25T18:47:04.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.030+0000 7f5f67b9d640 1 -- start start 2024-03-25T18:47:04.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.030+0000 7f5f67b9d640 10 monclient: init 2024-03-25T18:47:04.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:04.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:47:04.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f600b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:04.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f600b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:04.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6014be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:04.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600af940 con 0x7f5f600b4d60 2024-03-25T18:47:04.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600b0a70 con 0x7f5f6014be90 2024-03-25T18:47:04.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600b0d50 con 0x7f5f600b1730 2024-03-25T18:47:04.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:04.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.031+0000 7f5f67b9d640 10 monclient(hunting): authenticate will time out at 6530.220281s 2024-03-25T18:47:04.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.043+0000 7f5f6613a640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f5f6014be90 0x7f5f6014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:50294/0 (socket says 172.21.15.32:50294) 2024-03-25T18:47:04.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.043+0000 7f5f6613a640 1 -- 172.21.15.32:0/1905108361 learned_addr learned my addr 172.21.15.32:0/1905108361 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:04.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.057+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223123253 0 0) 0x7f5f600b0a70 con 0x7f5f6014be90 2024-03-25T18:47:04.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.057+0000 7f5f64937640 10 monclient(hunting): _negotiate 2024-03-25T18:47:04.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.057+0000 7f5f64937640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:04.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.057+0000 7f5f64937640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:04.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.057+0000 7f5f64937640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:47:04.057 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.057+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f4c003520 con 0x7f5f6014be90 2024-03-25T18:47:04.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.062+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972914733 0 0) 0x7f5f600af940 con 0x7f5f600b4d60 2024-03-25T18:47:04.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.062+0000 7f5f64937640 10 monclient(hunting): _negotiate 2024-03-25T18:47:04.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.062+0000 7f5f64937640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:04.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.062+0000 7f5f64937640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:04.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.062+0000 7f5f64937640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:47:04.062 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.062+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f600b0a70 con 0x7f5f600b4d60 2024-03-25T18:47:04.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.071+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3437284451 0 0) 0x7f5f600b0d50 con 0x7f5f600b1730 2024-03-25T18:47:04.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.071+0000 7f5f64937640 10 monclient(hunting): _negotiate 2024-03-25T18:47:04.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.071+0000 7f5f64937640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:04.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.071+0000 7f5f64937640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:04.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.071+0000 7f5f64937640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:47:04.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.071+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f600af940 con 0x7f5f600b1730 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1553399736 0 0) 0x7f5f600b0a70 con 0x7f5f600b4d60 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 >> v1:172.21.15.32:6790/0 conn(0x7f5f600b1730 legacy=0x7f5f600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 >> v1:172.21.15.136:6789/0 conn(0x7f5f6014be90 legacy=0x7f5f6014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 10 monclient: _finish_hunting 0 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 1 monclient: found mon.noname-a 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600b1030 con 0x7f5f600b4d60 2024-03-25T18:47:04.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 10 monclient: _finish_auth 0 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 10 monclient: _check_auth_tickets 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f5c001670 con 0x7f5f600b4d60 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.083+0000 7f5f64937640 10 monclient: dump: 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:04.085 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:04.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.084+0000 7f5f67b9d640 5 monclient: authenticate success, global_id 4584 2024-03-25T18:47:04.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.084+0000 7f5f67b9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:04.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.105+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f5c0017d0 con 0x7f5f600b4d60 2024-03-25T18:47:04.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.105+0000 7f5f64937640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:04.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.105+0000 7f5f64937640 1 -- 172.21.15.32:0/1905108361 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f5c001be0 con 0x7f5f600b4d60 2024-03-25T18:47:04.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.105+0000 7f5f64937640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:04.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.105+0000 7f5f64937640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:04.105 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.105+0000 7f5f64937640 10 monclient: dump: 2024-03-25T18:47:04.105 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:04.105 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.105+0000 7f5f67b9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:47:04.106 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.105+0000 7f5f67b9d640 4 set_mon_vals no callback set 2024-03-25T18:47:04.107 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.106+0000 7f5f67b9d640 10 monclient: shutdown 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.106+0000 7f5f67b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.106+0000 7f5f67b9d640 1 -- 172.21.15.32:0/1905108361 >> v1:172.21.15.32:6789/0 conn(0x7f5f600b4d60 legacy=0x7f5f6013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.106+0000 7f5f67b9d640 1 -- 172.21.15.32:0/1905108361 shutdown_connections 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.106+0000 7f5f67b9d640 1 -- 172.21.15.32:0/1905108361 >> 172.21.15.32:0/1905108361 conn(0x7f5f600617a0 msgr2=0x7f5f60061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.106+0000 7f5f67b9d640 1 -- 172.21.15.32:0/1905108361 shutdown_connections 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.106+0000 7f5f67b9d640 1 -- 172.21.15.32:0/1905108361 wait complete. 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.107+0000 7f5f67b9d640 10 monclient: build_initial_monmap 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.107+0000 7f5f67b9d640 10 monclient: monmap: 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:47:04.108 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.107+0000 7f5f67b9d640 1 librados: starting msgr at 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.107+0000 7f5f67b9d640 1 librados: starting objecter 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 1 Processor -- start 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 1 -- start start 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 1 librados: setting wanted keys 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 1 librados: calling monclient init 2024-03-25T18:47:04.109 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 10 monclient: init 2024-03-25T18:47:04.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:04.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:47:04.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 10 monclient(hunting): picked mon.noname-a con 0x7f5f600b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:04.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.108+0000 7f5f67b9d640 10 monclient(hunting): picked mon.noname-b con 0x7f5f600b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:04.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.109+0000 7f5f67b9d640 10 monclient(hunting): picked mon.noname-c con 0x7f5f6014be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:04.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.109+0000 7f5f67b9d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f60143d40 con 0x7f5f600b1730 2024-03-25T18:47:04.110 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.109+0000 7f5f67b9d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f60140f00 con 0x7f5f600b4d60 2024-03-25T18:47:04.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.109+0000 7f5f67b9d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f60138460 con 0x7f5f6014be90 2024-03-25T18:47:04.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.109+0000 7f5f67b9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:04.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.109+0000 7f5f67b9d640 10 monclient(hunting): authenticate will time out at 6530.297673s 2024-03-25T18:47:04.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.115+0000 7f5f6613a640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f5f6014be90 0x7f5f601d7d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:33560/0 (socket says 172.21.15.32:33560) 2024-03-25T18:47:04.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.115+0000 7f5f6613a640 1 -- 172.21.15.32:0/4411495 learned_addr learned my addr 172.21.15.32:0/4411495 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:04.116 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.116+0000 7f5f567fc640 10 client.?.objecter ms_handle_connect 0x7f5f6014be90 2024-03-25T18:47:04.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.116+0000 7f5f567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:04.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.120+0000 7f5f567fc640 10 client.?.objecter ms_handle_connect 0x7f5f600b4d60 2024-03-25T18:47:04.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.120+0000 7f5f567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:04.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.123+0000 7f5f567fc640 10 client.?.objecter ms_handle_connect 0x7f5f600b1730 2024-03-25T18:47:04.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.123+0000 7f5f567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:04.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.134+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1642617082 0 0) 0x7f5f60143d40 con 0x7f5f600b1730 2024-03-25T18:47:04.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.134+0000 7f5f567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:04.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.134+0000 7f5f567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:04.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.134+0000 7f5f567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:04.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.134+0000 7f5f567fc640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:47:04.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.134+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f44004eb0 con 0x7f5f600b1730 2024-03-25T18:47:04.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.135+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 813710508 0 0) 0x7f5f60140f00 con 0x7f5f600b4d60 2024-03-25T18:47:04.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.135+0000 7f5f567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:04.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.135+0000 7f5f567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:04.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.135+0000 7f5f567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:04.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.135+0000 7f5f567fc640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:47:04.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.135+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f60143d40 con 0x7f5f600b4d60 2024-03-25T18:47:04.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.142+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 954683491 0 0) 0x7f5f60138460 con 0x7f5f6014be90 2024-03-25T18:47:04.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.142+0000 7f5f567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:04.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.142+0000 7f5f567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:04.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.142+0000 7f5f567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:04.142 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.142+0000 7f5f567fc640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:47:04.143 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.142+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f60140f00 con 0x7f5f6014be90 2024-03-25T18:47:04.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.147+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4123324539 0 0) 0x7f5f60143d40 con 0x7f5f600b4d60 2024-03-25T18:47:04.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.147+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f60138460 con 0x7f5f600b4d60 2024-03-25T18:47:04.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.147+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f50001870 con 0x7f5f600b4d60 2024-03-25T18:47:04.147 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.147+0000 7f5f567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.147+0000 7f5f567fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.147+0000 7f5f567fc640 10 monclient(hunting): dump: 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:04.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.147+0000 7f5f567fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.148+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1042363262 0 0) 0x7f5f44004eb0 con 0x7f5f600b1730 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.148+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f60143d40 con 0x7f5f600b1730 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.148+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f5c001ba0 con 0x7f5f600b1730 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.148+0000 7f5f567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.148+0000 7f5f567fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.148+0000 7f5f567fc640 10 monclient(hunting): dump: 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.149 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.150 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:04.150 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:04.150 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:04.150 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:04.150 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:04.150 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:04.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2940429358 0 0) 0x7f5f60138460 con 0x7f5f600b4d60 2024-03-25T18:47:04.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 >> v1:172.21.15.32:6790/0 conn(0x7f5f6014be90 legacy=0x7f5f601d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:04.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 >> v1:172.21.15.32:6789/0 conn(0x7f5f600b1730 legacy=0x7f5f60137c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:04.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:47:04.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 1 monclient: found mon.b 2024-03-25T18:47:04.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:04.166 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f60141130 con 0x7f5f600b4d60 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 10 monclient: _finish_auth 0 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 10 monclient: _renew_subs 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f60137660 con 0x7f5f600b4d60 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 10 monclient: _renew_subs 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f60143030 con 0x7f5f600b4d60 2024-03-25T18:47:04.167 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.165+0000 7f5f67b9d640 1 librados: init done 2024-03-25T18:47:04.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.166+0000 7f5f67b9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:04.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.166+0000 7f5f67b9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:04.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.166+0000 7f5f67b9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:04.169 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.166+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f600627b0 con 0x7f5f600b4d60 2024-03-25T18:47:04.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.434+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f50003610 con 0x7f5f600b4d60 2024-03-25T18:47:04.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.434+0000 7f5f567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.434+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f50001d80 con 0x7f5f600b4d60 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.434+0000 7f5f567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.434+0000 7f5f567fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.434+0000 7f5f567fc640 10 monclient: dump: 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:04.435 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:04.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.434+0000 7f5f57fff640 4 set_mon_vals no callback set 2024-03-25T18:47:04.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.435+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5f50003b20 con 0x7f5f600b4d60 2024-03-25T18:47:04.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.435+0000 7f5f65939640 10 monclient: get_auth_request con 0x7f5f4406df70 auth_method 0 2024-03-25T18:47:04.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.436+0000 7f5f567fc640 10 client.4564.objecter ms_handle_connect 0x7f5f4406df70 2024-03-25T18:47:04.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.440+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5f50053520 con 0x7f5f600b4d60 2024-03-25T18:47:04.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.440+0000 7f5f567fc640 10 client.4564.objecter ms_dispatch 0x7f5f60000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:47:04.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.440+0000 7f5f567fc640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:47:04.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.440+0000 7f5f567fc640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:47:04.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.440+0000 7f5f567fc640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:47:04.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.441+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f5004f030 con 0x7f5f600b4d60 2024-03-25T18:47:04.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.441+0000 7f5f567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:04.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.441+0000 7f5f567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:47:04.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.673+0000 7f5f67b9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:04.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.673+0000 7f5f67b9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:04.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.673+0000 7f5f67b9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:04.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.673+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f601d7980 con 0x7f5f600b4d60 2024-03-25T18:47:04.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.702+0000 7f5f567fc640 1 -- 172.21.15.32:0/4411495 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5f50050f60 con 0x7f5f600b4d60 2024-03-25T18:47:04.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.702+0000 7f5f567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:04.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.702+0000 7f5f567fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:47:04.703 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:47:04.703 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 10 librados: watch_flush enter 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 10 librados: watch_flush exit 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 >> v1:172.21.15.136:6800/1141502808 conn(0x7f5f4406df70 legacy=0x7f5f44070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 10 monclient: shutdown 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 >> v1:172.21.15.136:6789/0 conn(0x7f5f600b4d60 legacy=0x7f5f60142920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 shutdown_connections 2024-03-25T18:47:04.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 >> 172.21.15.32:0/4411495 conn(0x7f5f600617a0 msgr2=0x7f5f600b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:04.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 shutdown_connections 2024-03-25T18:47:04.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 1 -- 172.21.15.32:0/4411495 wait complete. 2024-03-25T18:47:04.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:04.705+0000 7f5f67b9d640 1 librados: shutdown 2024-03-25T18:47:04.717 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:47:08.490 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:20.818 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:08.495 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:20.823 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:08.524 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:20.852 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:08.550 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:20.867 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:08.564 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:20.881 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:08.572 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:20.900 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:08.587 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:20.915 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:08.595 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:20.912 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:08.618 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:20.946 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:11.524 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:47:11.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.638+0000 7fd3d601e640 10 monclient: get_monmap_and_config 2024-03-25T18:47:11.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.638+0000 7fd3d601e640 10 monclient: build_initial_monmap 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.638+0000 7fd3d601e640 10 monclient: monmap: 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:47:11.638988+0000 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:47:11.638988+0000 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:47:11.639 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:11.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 1 Processor -- start 2024-03-25T18:47:11.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 1 -- start start 2024-03-25T18:47:11.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 10 monclient: init 2024-03-25T18:47:11.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:11.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:47:11.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d00a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:11.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:11.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3d01430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:11.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.639+0000 7fd3d601e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d0142520 con 0x7fd3d00a98f0 2024-03-25T18:47:11.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.640+0000 7fd3d601e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d00b28a0 con 0x7fd3d013f710 2024-03-25T18:47:11.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.640+0000 7fd3d601e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d00b2a30 con 0x7fd3d01430c0 2024-03-25T18:47:11.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.640+0000 7fd3d601e640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:11.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.640+0000 7fd3d601e640 10 monclient(hunting): authenticate will time out at 6537.828559s 2024-03-25T18:47:11.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.648+0000 7fd3cf7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd3d00a98f0 0x7fd3d00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44702/0 (socket says 172.21.15.32:44702) 2024-03-25T18:47:11.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.648+0000 7fd3cf7fe640 1 -- 172.21.15.32:0/3768836156 learned_addr learned my addr 172.21.15.32:0/3768836156 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.664+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255846050 0 0) 0x7fd3d00b28a0 con 0x7fd3d013f710 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.664+0000 7fd3ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.664+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.664+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.664+0000 7fd3ce7fc640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.664+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3b8003520 con 0x7fd3d013f710 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.665+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1631141450 0 0) 0x7fd3d0142520 con 0x7fd3d00a98f0 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.665+0000 7fd3ce7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.665+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.665+0000 7fd3ce7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:11.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.665+0000 7fd3ce7fc640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:47:11.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.665+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d00b28a0 con 0x7fd3d00a98f0 2024-03-25T18:47:11.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1179613798 0 0) 0x7fd3b8003520 con 0x7fd3d013f710 2024-03-25T18:47:11.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 >> v1:172.21.15.32:6790/0 conn(0x7fd3d01430c0 legacy=0x7fd3d014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:11.672 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 >> v1:172.21.15.32:6789/0 conn(0x7fd3d00a98f0 legacy=0x7fd3d00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 1 monclient: found mon.noname-c 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d00b44c0 con 0x7fd3d013f710 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 10 monclient: _finish_auth 0 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd3c4001690 con 0x7fd3d013f710 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3ce7fc640 10 monclient: dump: 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:11.673 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3d601e640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:47:11.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.672+0000 7fd3d601e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:11.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.682+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3c40017f0 con 0x7fd3d013f710 2024-03-25T18:47:11.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.682+0000 7fd3ce7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:11.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.682+0000 7fd3ce7fc640 1 -- 172.21.15.32:0/3768836156 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd3c4001c00 con 0x7fd3d013f710 2024-03-25T18:47:11.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.682+0000 7fd3d601e640 10 monclient: get_monmap_and_config success 2024-03-25T18:47:11.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.682+0000 7fd3d601e640 4 set_mon_vals no callback set 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.682+0000 7fd3ce7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.682+0000 7fd3ce7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.682+0000 7fd3ce7fc640 10 monclient: dump: 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:11.683 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:11.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.683+0000 7fd3d601e640 10 monclient: shutdown 2024-03-25T18:47:11.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.683+0000 7fd3d601e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:11.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.683+0000 7fd3d601e640 1 -- 172.21.15.32:0/3768836156 >> v1:172.21.15.136:6789/0 conn(0x7fd3d013f710 legacy=0x7fd3d01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:11.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.683+0000 7fd3d601e640 1 -- 172.21.15.32:0/3768836156 shutdown_connections 2024-03-25T18:47:11.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.683+0000 7fd3d601e640 1 -- 172.21.15.32:0/3768836156 >> 172.21.15.32:0/3768836156 conn(0x7fd3d00b7ea0 msgr2=0x7fd3d00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:11.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.683+0000 7fd3d601e640 1 -- 172.21.15.32:0/3768836156 shutdown_connections 2024-03-25T18:47:11.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.683+0000 7fd3d601e640 1 -- 172.21.15.32:0/3768836156 wait complete. 2024-03-25T18:47:11.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.684+0000 7fd3d601e640 10 monclient: build_initial_monmap 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.684+0000 7fd3d601e640 10 monclient: monmap: 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.684+0000 7fd3d601e640 1 librados: starting msgr at 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.684+0000 7fd3d601e640 1 librados: starting objecter 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.684+0000 7fd3d601e640 1 Processor -- start 2024-03-25T18:47:11.686 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 1 -- start start 2024-03-25T18:47:11.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 1 librados: setting wanted keys 2024-03-25T18:47:11.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 1 librados: calling monclient init 2024-03-25T18:47:11.687 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 10 monclient: init 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 10 monclient(hunting): picked mon.noname-b con 0x7fd3d00a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 10 monclient(hunting): picked mon.noname-a con 0x7fd3d013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 10 monclient(hunting): picked mon.noname-c con 0x7fd3d01430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d0134a40 con 0x7fd3d013f710 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d006fad0 con 0x7fd3d00a98f0 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3d006f270 con 0x7fd3d01430c0 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:11.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.685+0000 7fd3d601e640 10 monclient(hunting): authenticate will time out at 6537.874465s 2024-03-25T18:47:11.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.690+0000 7fd3cf7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fd3d00a98f0 0x7fd3d0070a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:56408/0 (socket says 172.21.15.32:56408) 2024-03-25T18:47:11.690 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.690+0000 7fd3cf7fe640 1 -- 172.21.15.32:0/2001989968 learned_addr learned my addr 172.21.15.32:0/2001989968 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:11.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.691+0000 7fd3b7fff640 10 client.?.objecter ms_handle_connect 0x7fd3d00a98f0 2024-03-25T18:47:11.691 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.691+0000 7fd3b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:11.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.701+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1334035081 0 0) 0x7fd3d006fad0 con 0x7fd3d00a98f0 2024-03-25T18:47:11.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.701+0000 7fd3b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:47:11.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.701+0000 7fd3b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:11.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.701+0000 7fd3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:11.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.701+0000 7fd3b7fff640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:47:11.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.701+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3bc004eb0 con 0x7fd3d00a98f0 2024-03-25T18:47:11.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.705+0000 7fd3b7fff640 10 client.?.objecter ms_handle_connect 0x7fd3d01430c0 2024-03-25T18:47:11.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.705+0000 7fd3b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:11.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.709+0000 7fd3b7fff640 10 client.?.objecter ms_handle_connect 0x7fd3d013f710 2024-03-25T18:47:11.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.709+0000 7fd3b7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:11.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.718+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449533460 0 0) 0x7fd3d0134a40 con 0x7fd3d013f710 2024-03-25T18:47:11.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.718+0000 7fd3b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:47:11.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.718+0000 7fd3b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:11.718 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.718+0000 7fd3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:11.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.718+0000 7fd3b7fff640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:47:11.719 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.718+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d006fad0 con 0x7fd3d013f710 2024-03-25T18:47:11.720 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.720+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2526934326 0 0) 0x7fd3d006f270 con 0x7fd3d01430c0 2024-03-25T18:47:11.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.720+0000 7fd3b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:47:11.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.720+0000 7fd3b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:11.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.720+0000 7fd3b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:11.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.720+0000 7fd3b7fff640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:47:11.721 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.720+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d0134a40 con 0x7fd3d01430c0 2024-03-25T18:47:11.730 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.730+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4086259990 0 0) 0x7fd3bc004eb0 con 0x7fd3d00a98f0 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.730+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3d006f270 con 0x7fd3d00a98f0 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.730+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd3c00019b0 con 0x7fd3d00a98f0 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.730+0000 7fd3b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.730+0000 7fd3b7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.730+0000 7fd3b7fff640 10 monclient(hunting): dump: 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:11.731 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:11.732 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.730+0000 7fd3b7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:47:11.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 830437721 0 0) 0x7fd3d006fad0 con 0x7fd3d013f710 2024-03-25T18:47:11.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3bc004eb0 con 0x7fd3d013f710 2024-03-25T18:47:11.745 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd3c4001b40 con 0x7fd3d013f710 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 10 monclient(hunting): dump: 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:11.746 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3004538240 0 0) 0x7fd3d0134a40 con 0x7fd3d01430c0 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3d006fad0 con 0x7fd3d01430c0 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd3c8001a60 con 0x7fd3d01430c0 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.744+0000 7fd3b7fff640 10 monclient(hunting): dump: 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:11.747 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2564039714 0 0) 0x7fd3d006f270 con 0x7fd3d00a98f0 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 >> v1:172.21.15.32:6790/0 conn(0x7fd3d01430c0 legacy=0x7fd3d006e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 >> v1:172.21.15.32:6789/0 conn(0x7fd3d013f710 legacy=0x7fd3d006dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 1 monclient: found mon.b 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d006fd40 con 0x7fd3d00a98f0 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 5 monclient: authenticate success, global_id 4570 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 10 monclient: _renew_subs 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:11.748 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3d0070020 con 0x7fd3d00a98f0 2024-03-25T18:47:11.749 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 10 client.4570.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:47:11.749 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 10 monclient: _renew_subs 2024-03-25T18:47:11.749 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:11.749 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3d006ecc0 con 0x7fd3d00a98f0 2024-03-25T18:47:11.749 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.745+0000 7fd3d601e640 1 librados: init done 2024-03-25T18:47:11.749 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.746+0000 7fd3d601e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:11.750 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.746+0000 7fd3d601e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:11.750 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.746+0000 7fd3d601e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:11.750 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:11.746+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3d00627b0 con 0x7fd3d00a98f0 2024-03-25T18:47:12.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.027+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3c00047e0 con 0x7fd3d00a98f0 2024-03-25T18:47:12.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.027+0000 7fd3b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:12.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.027+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd3c0004af0 con 0x7fd3d00a98f0 2024-03-25T18:47:12.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.027+0000 7fd3b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:12.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.027+0000 7fd3b7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:12.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.027+0000 7fd3b7fff640 10 monclient: dump: 2024-03-25T18:47:12.027 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:12.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.027+0000 7fd3cdffb640 4 set_mon_vals no callback set 2024-03-25T18:47:12.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.027+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fd3c001ab00 con 0x7fd3d00a98f0 2024-03-25T18:47:12.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.028+0000 7fd3ceffd640 10 monclient: get_auth_request con 0x7fd3bc06e0f0 auth_method 0 2024-03-25T18:47:12.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.028+0000 7fd3b7fff640 10 client.4570.objecter ms_handle_connect 0x7fd3bc06e0f0 2024-03-25T18:47:12.032 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.032+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fd3c0054850 con 0x7fd3d00a98f0 2024-03-25T18:47:12.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.032+0000 7fd3b7fff640 10 client.4570.objecter ms_dispatch 0x7fd3d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:47:12.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.032+0000 7fd3b7fff640 3 client.4570.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:47:12.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.032+0000 7fd3b7fff640 3 client.4570.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:47:12.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.032+0000 7fd3b7fff640 20 client.4570.objecter dump_active .. 0 homeless 2024-03-25T18:47:12.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.033+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3c0050190 con 0x7fd3d00a98f0 2024-03-25T18:47:12.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.033+0000 7fd3b7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:12.033 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.033+0000 7fd3b7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:47:12.295 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.295+0000 7fd3d601e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:12.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.295+0000 7fd3d601e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:12.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.295+0000 7fd3d601e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:12.296 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.295+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3d01421d0 con 0x7fd3d00a98f0 2024-03-25T18:47:12.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.403+0000 7fd3b7fff640 1 -- 172.21.15.32:0/2001989968 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fd3c0052290 con 0x7fd3d00a98f0 2024-03-25T18:47:12.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.403+0000 7fd3b7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:12.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.403+0000 7fd3b7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:47:12.404 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:47:12.404 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 10 librados: watch_flush enter 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 10 librados: watch_flush exit 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 20 client.4570.objecter shutdown clearing up homeless session... 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 10 client.4570.objecter successfully canceled tick 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 >> v1:172.21.15.136:6800/1141502808 conn(0x7fd3bc06e0f0 legacy=0x7fd3bc0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 10 monclient: shutdown 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 >> v1:172.21.15.136:6789/0 conn(0x7fd3d00a98f0 legacy=0x7fd3d0070a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 shutdown_connections 2024-03-25T18:47:12.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 >> 172.21.15.32:0/2001989968 conn(0x7fd3d00b7ea0 msgr2=0x7fd3d0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:12.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 shutdown_connections 2024-03-25T18:47:12.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 1 -- 172.21.15.32:0/2001989968 wait complete. 2024-03-25T18:47:12.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:12.405+0000 7fd3d601e640 1 librados: shutdown 2024-03-25T18:47:12.416 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:47:19.222 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:47:19.338 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.338+0000 7f1c6b359640 10 monclient: get_monmap_and_config 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.338+0000 7f1c6b359640 10 monclient: build_initial_monmap 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.338+0000 7f1c6b359640 10 monclient: monmap: 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:47:19.339148+0000 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:47:19.339148+0000 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:19.339 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:47:19.340 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:47:19.340 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.339+0000 7f1c6b359640 1 Processor -- start 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.339+0000 7f1c6b359640 1 -- start start 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.339+0000 7f1c6b359640 10 monclient: init 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 10 monclient(hunting): picked mon.noname-b con 0x7f1c640a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 10 monclient(hunting): picked mon.noname-c con 0x7f1c6413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 10 monclient(hunting): picked mon.noname-a con 0x7f1c641430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c64142520 con 0x7f1c641430c0 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640b28a0 con 0x7f1c6413f710 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c640b2a30 con 0x7f1c640a98f0 2024-03-25T18:47:19.341 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:19.342 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.340+0000 7f1c6b359640 10 monclient(hunting): authenticate will time out at 6545.529082s 2024-03-25T18:47:19.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.350+0000 7f1c688f4640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1c6413f710 0x7f1c641429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43130/0 (socket says 172.21.15.32:43130) 2024-03-25T18:47:19.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.350+0000 7f1c688f4640 1 -- 172.21.15.32:0/1999042099 learned_addr learned my addr 172.21.15.32:0/1999042099 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:19.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106293470 0 0) 0x7f1c64142520 con 0x7f1c641430c0 2024-03-25T18:47:19.365 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c50003520 con 0x7f1c641430c0 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824135045 0 0) 0x7f1c640b28a0 con 0x7f1c6413f710 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:47:19.366 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.365+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c64142520 con 0x7f1c6413f710 2024-03-25T18:47:19.374 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2172569163 0 0) 0x7f1c64142520 con 0x7f1c6413f710 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 >> v1:172.21.15.32:6790/0 conn(0x7f1c640a98f0 legacy=0x7f1c640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 >> v1:172.21.15.32:6789/0 conn(0x7f1c641430c0 legacy=0x7f1c6414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 1 monclient: found mon.noname-c 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640b44c0 con 0x7f1c6413f710 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 10 monclient: _finish_auth 0 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c54001670 con 0x7f1c6413f710 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:47:19.375 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c5bfff640 10 monclient: dump: 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c6b359640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:47:19.376 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.374+0000 7f1c6b359640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:19.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.386+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c540017d0 con 0x7f1c6413f710 2024-03-25T18:47:19.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.386+0000 7f1c5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:19.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.386+0000 7f1c5bfff640 1 -- 172.21.15.32:0/1999042099 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c54001ba0 con 0x7f1c6413f710 2024-03-25T18:47:19.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.386+0000 7f1c5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.386+0000 7f1c5bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.386+0000 7f1c5bfff640 10 monclient: dump: 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:19.387 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.386+0000 7f1c6b359640 10 monclient: get_monmap_and_config success 2024-03-25T18:47:19.388 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.386+0000 7f1c6b359640 4 set_mon_vals no callback set 2024-03-25T18:47:19.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.387+0000 7f1c6b359640 10 monclient: shutdown 2024-03-25T18:47:19.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.387+0000 7f1c6b359640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:19.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.387+0000 7f1c6b359640 1 -- 172.21.15.32:0/1999042099 >> v1:172.21.15.136:6789/0 conn(0x7f1c6413f710 legacy=0x7f1c641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:19.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.387+0000 7f1c6b359640 1 -- 172.21.15.32:0/1999042099 shutdown_connections 2024-03-25T18:47:19.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.387+0000 7f1c6b359640 1 -- 172.21.15.32:0/1999042099 >> 172.21.15.32:0/1999042099 conn(0x7f1c640b7ea0 msgr2=0x7f1c640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:19.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.387+0000 7f1c6b359640 1 -- 172.21.15.32:0/1999042099 shutdown_connections 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.387+0000 7f1c6b359640 1 -- 172.21.15.32:0/1999042099 wait complete. 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.388+0000 7f1c6b359640 10 monclient: build_initial_monmap 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.388+0000 7f1c6b359640 10 monclient: monmap: 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:19.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.388+0000 7f1c6b359640 1 librados: starting msgr at 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.388+0000 7f1c6b359640 1 librados: starting objecter 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 1 Processor -- start 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 1 -- start start 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 1 librados: setting wanted keys 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 1 librados: calling monclient init 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 10 monclient: init 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 10 monclient(hunting): picked mon.noname-c con 0x7f1c640a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 10 monclient(hunting): picked mon.noname-b con 0x7f1c6413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.389+0000 7f1c6b359640 10 monclient(hunting): picked mon.noname-a con 0x7f1c641430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.390+0000 7f1c6b359640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6413b260 con 0x7f1c641430c0 2024-03-25T18:47:19.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.390+0000 7f1c6b359640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6413b540 con 0x7f1c6413f710 2024-03-25T18:47:19.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.390+0000 7f1c6b359640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c6413a100 con 0x7f1c640a98f0 2024-03-25T18:47:19.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.390+0000 7f1c6b359640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:19.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.390+0000 7f1c6b359640 10 monclient(hunting): authenticate will time out at 6545.578594s 2024-03-25T18:47:19.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.397+0000 7f1c688f4640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f1c6413f710 0x7f1c641df270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43166/0 (socket says 172.21.15.32:43166) 2024-03-25T18:47:19.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.397+0000 7f1c688f4640 1 -- 172.21.15.32:0/2054180589 learned_addr learned my addr 172.21.15.32:0/2054180589 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:19.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.398+0000 7f1c59ffb640 10 client.?.objecter ms_handle_connect 0x7f1c6413f710 2024-03-25T18:47:19.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.398+0000 7f1c59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:19.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.404+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3389103452 0 0) 0x7f1c6413b540 con 0x7f1c6413f710 2024-03-25T18:47:19.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.404+0000 7f1c59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:47:19.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.404+0000 7f1c59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:19.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.404+0000 7f1c59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:19.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.404+0000 7f1c59ffb640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:47:19.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.404+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1c48004eb0 con 0x7f1c6413f710 2024-03-25T18:47:19.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.405+0000 7f1c59ffb640 10 client.?.objecter ms_handle_connect 0x7f1c641430c0 2024-03-25T18:47:19.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.405+0000 7f1c59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:19.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.411+0000 7f1c59ffb640 10 client.?.objecter ms_handle_connect 0x7f1c640a98f0 2024-03-25T18:47:19.411 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.411+0000 7f1c59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:19.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.413+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2355979412 0 0) 0x7f1c48004eb0 con 0x7f1c6413f710 2024-03-25T18:47:19.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.413+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1c6413b540 con 0x7f1c6413f710 2024-03-25T18:47:19.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.413+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c54001ba0 con 0x7f1c6413f710 2024-03-25T18:47:19.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.413+0000 7f1c59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:19.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.413+0000 7f1c59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:47:19.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.413+0000 7f1c59ffb640 10 monclient(hunting): dump: 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:19.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.413+0000 7f1c59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3062567822 0 0) 0x7f1c6413b540 con 0x7f1c6413f710 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 >> v1:172.21.15.32:6790/0 conn(0x7f1c640a98f0 legacy=0x7f1c640b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 >> v1:172.21.15.32:6789/0 conn(0x7f1c641430c0 legacy=0x7f1c641e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 1 monclient: found mon.b 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c640b7770 con 0x7f1c6413f710 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 10 monclient: _finish_auth 0 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:47:19.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 5 monclient: authenticate success, global_id 4576 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 10 monclient: _renew_subs 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c64139d20 con 0x7f1c6413f710 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 10 client.4576.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 10 monclient: _renew_subs 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c640b7ca0 con 0x7f1c6413f710 2024-03-25T18:47:19.422 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.420+0000 7f1c6b359640 1 librados: init done 2024-03-25T18:47:19.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.421+0000 7f1c6b359640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:19.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.421+0000 7f1c6b359640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:19.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.421+0000 7f1c6b359640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:19.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.421+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c640b3ef0 con 0x7f1c6413f710 2024-03-25T18:47:19.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.440+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1c54004a90 con 0x7f1c6413f710 2024-03-25T18:47:19.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.440+0000 7f1c59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:19.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.440+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1c54004d00 con 0x7f1c6413f710 2024-03-25T18:47:19.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.440+0000 7f1c59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.440+0000 7f1c59ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.440+0000 7f1c59ffb640 10 monclient: dump: 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:19.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.440+0000 7f1c5b7fe640 4 set_mon_vals no callback set 2024-03-25T18:47:19.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.441+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f1c54004f20 con 0x7f1c6413f710 2024-03-25T18:47:19.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.441+0000 7f1c690f5640 10 monclient: get_auth_request con 0x7f1c4806dce0 auth_method 0 2024-03-25T18:47:19.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.442+0000 7f1c59ffb640 10 client.4576.objecter ms_handle_connect 0x7f1c4806dce0 2024-03-25T18:47:19.444 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.443+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f1c54054870 con 0x7f1c6413f710 2024-03-25T18:47:19.444 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.443+0000 7f1c59ffb640 10 client.4576.objecter ms_dispatch 0x7f1c64000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:47:19.444 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.443+0000 7f1c59ffb640 3 client.4576.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:47:19.444 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.443+0000 7f1c59ffb640 3 client.4576.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:47:19.444 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.443+0000 7f1c59ffb640 20 client.4576.objecter dump_active .. 0 homeless 2024-03-25T18:47:19.714 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.714+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1c54050380 con 0x7f1c6413f710 2024-03-25T18:47:19.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.714+0000 7f1c59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:19.715 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.714+0000 7f1c59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:47:19.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.963+0000 7f1c6b359640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:19.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.963+0000 7f1c6b359640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:19.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.963+0000 7f1c6b359640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:19.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.963+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1c6413a290 con 0x7f1c6413f710 2024-03-25T18:47:19.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.981+0000 7f1c59ffb640 1 -- 172.21.15.32:0/2054180589 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f1c540522b0 con 0x7f1c6413f710 2024-03-25T18:47:19.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.981+0000 7f1c59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:19.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.981+0000 7f1c59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:47:19.982 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:47:19.982 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:47:19.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 10 librados: watch_flush enter 2024-03-25T18:47:19.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 10 librados: watch_flush exit 2024-03-25T18:47:19.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 20 client.4576.objecter shutdown clearing up homeless session... 2024-03-25T18:47:19.984 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 10 client.4576.objecter successfully canceled tick 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 >> v1:172.21.15.136:6800/1141502808 conn(0x7f1c4806dce0 legacy=0x7f1c480701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 10 monclient: shutdown 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 >> v1:172.21.15.136:6789/0 conn(0x7f1c6413f710 legacy=0x7f1c641df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 shutdown_connections 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 >> 172.21.15.32:0/2054180589 conn(0x7f1c640b7ea0 msgr2=0x7f1c64074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 shutdown_connections 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 1 -- 172.21.15.32:0/2054180589 wait complete. 2024-03-25T18:47:19.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:19.983+0000 7f1c6b359640 1 librados: shutdown 2024-03-25T18:47:19.994 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:47:26.801 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.917+0000 7fb573893640 10 monclient: get_monmap_and_config 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.917+0000 7fb573893640 10 monclient: build_initial_monmap 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.917+0000 7fb573893640 10 monclient: monmap: 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:47:26.919254+0000 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:47:26.919254+0000 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:47:26.919 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.918+0000 7fb573893640 1 Processor -- start 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.918+0000 7fb573893640 1 -- start start 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.918+0000 7fb573893640 10 monclient: init 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 10 monclient(hunting): picked mon.noname-c con 0x7fb56c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 10 monclient(hunting): picked mon.noname-a con 0x7fb56c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 10 monclient(hunting): picked mon.noname-b con 0x7fb56c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c142520 con 0x7fb56c13f710 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0b28a0 con 0x7fb56c0a98f0 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0b2a30 con 0x7fb56c1430c0 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:26.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.919+0000 7fb573893640 10 monclient(hunting): authenticate will time out at 6553.109224s 2024-03-25T18:47:26.926 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.925+0000 7fb571e30640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fb56c1430c0 0x7fb56c14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:36628/0 (socket says 172.21.15.32:36628) 2024-03-25T18:47:26.926 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.925+0000 7fb571e30640 1 -- 172.21.15.32:0/3983550538 learned_addr learned my addr 172.21.15.32:0/3983550538 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:26.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.944+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089808766 0 0) 0x7fb56c142520 con 0x7fb56c13f710 2024-03-25T18:47:26.945 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.944+0000 7fb563fff640 10 monclient(hunting): _negotiate 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.944+0000 7fb563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.944+0000 7fb563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.944+0000 7fb563fff640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.944+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb558003520 con 0x7fb56c13f710 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.945+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897991546 0 0) 0x7fb56c0b2a30 con 0x7fb56c1430c0 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.945+0000 7fb563fff640 10 monclient(hunting): _negotiate 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.945+0000 7fb563fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.945+0000 7fb563fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:26.946 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.945+0000 7fb563fff640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:47:26.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.945+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb56c142520 con 0x7fb56c1430c0 2024-03-25T18:47:26.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1047888930 0 0) 0x7fb558003520 con 0x7fb56c13f710 2024-03-25T18:47:26.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 >> v1:172.21.15.32:6790/0 conn(0x7fb56c1430c0 legacy=0x7fb56c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:26.962 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 >> v1:172.21.15.136:6789/0 conn(0x7fb56c0a98f0 legacy=0x7fb56c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 10 monclient: _finish_hunting 0 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 1 monclient: found mon.noname-a 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c0b44c0 con 0x7fb56c13f710 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 10 monclient: _finish_auth 0 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 10 monclient: _check_auth_tickets 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb55c001690 con 0x7fb56c13f710 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb573893640 5 monclient: authenticate success, global_id 4602 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb573893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb563fff640 10 monclient: dump: 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:26.963 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:26.964 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:26.964 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:26.964 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:26.964 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:26.964 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:26.964 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:26.964 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:26.964 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.961+0000 7fb573893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:26.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.984+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb55c0017f0 con 0x7fb56c13f710 2024-03-25T18:47:26.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.984+0000 7fb563fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:26.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.984+0000 7fb563fff640 1 -- 172.21.15.32:0/3983550538 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb55c001c00 con 0x7fb56c13f710 2024-03-25T18:47:26.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.984+0000 7fb563fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:26.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.984+0000 7fb563fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:26.985 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.984+0000 7fb563fff640 10 monclient: dump: 2024-03-25T18:47:26.985 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.984+0000 7fb573893640 10 monclient: get_monmap_and_config success 2024-03-25T18:47:26.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.984+0000 7fb573893640 4 set_mon_vals no callback set 2024-03-25T18:47:26.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.985+0000 7fb573893640 10 monclient: shutdown 2024-03-25T18:47:26.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.985+0000 7fb573893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:26.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.985+0000 7fb573893640 1 -- 172.21.15.32:0/3983550538 >> v1:172.21.15.32:6789/0 conn(0x7fb56c13f710 legacy=0x7fb56c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:26.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.985+0000 7fb573893640 1 -- 172.21.15.32:0/3983550538 shutdown_connections 2024-03-25T18:47:26.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.985+0000 7fb573893640 1 -- 172.21.15.32:0/3983550538 >> 172.21.15.32:0/3983550538 conn(0x7fb56c0b7ea0 msgr2=0x7fb56c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:26.987 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.985+0000 7fb573893640 1 -- 172.21.15.32:0/3983550538 shutdown_connections 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.985+0000 7fb573893640 1 -- 172.21.15.32:0/3983550538 wait complete. 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.986+0000 7fb573893640 10 monclient: build_initial_monmap 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.986+0000 7fb573893640 10 monclient: monmap: 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:47:26.988 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:26.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.987+0000 7fb573893640 1 librados: starting msgr at 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.987+0000 7fb573893640 1 librados: starting objecter 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.987+0000 7fb573893640 1 Processor -- start 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.987+0000 7fb573893640 1 -- start start 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.987+0000 7fb573893640 1 librados: setting wanted keys 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.987+0000 7fb573893640 1 librados: calling monclient init 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.987+0000 7fb573893640 10 monclient: init 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 10 monclient(hunting): picked mon.noname-c con 0x7fb56c0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 10 monclient(hunting): picked mon.noname-b con 0x7fb56c13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 10 monclient(hunting): picked mon.noname-a con 0x7fb56c1430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0abac0 con 0x7fb56c1430c0 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c0abda0 con 0x7fb56c13f710 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb56c1380e0 con 0x7fb56c0a98f0 2024-03-25T18:47:26.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:26.991 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:26.988+0000 7fb573893640 10 monclient(hunting): authenticate will time out at 6553.178511s 2024-03-25T18:47:27.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.000+0000 7fb570e2e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb56c13f710 0x7fb56c138b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:42146/0 (socket says 172.21.15.32:42146) 2024-03-25T18:47:27.001 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.000+0000 7fb570e2e640 1 -- 172.21.15.32:0/980220423 learned_addr learned my addr 172.21.15.32:0/980220423 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:27.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.004+0000 7fb561ffb640 10 client.?.objecter ms_handle_connect 0x7fb56c13f710 2024-03-25T18:47:27.005 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.004+0000 7fb561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:27.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.009+0000 7fb561ffb640 10 client.?.objecter ms_handle_connect 0x7fb56c0a98f0 2024-03-25T18:47:27.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.009+0000 7fb561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:27.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.013+0000 7fb561ffb640 10 client.?.objecter ms_handle_connect 0x7fb56c1430c0 2024-03-25T18:47:27.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.013+0000 7fb561ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:27.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.021+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103355825 0 0) 0x7fb56c0abda0 con 0x7fb56c13f710 2024-03-25T18:47:27.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.021+0000 7fb561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:47:27.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.021+0000 7fb561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:27.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.021+0000 7fb561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:27.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.021+0000 7fb561ffb640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:47:27.022 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.021+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb550004eb0 con 0x7fb56c13f710 2024-03-25T18:47:27.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.025+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1725782584 0 0) 0x7fb56c1380e0 con 0x7fb56c0a98f0 2024-03-25T18:47:27.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.025+0000 7fb561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:47:27.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.025+0000 7fb561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:27.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.025+0000 7fb561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:27.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.025+0000 7fb561ffb640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:47:27.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.025+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb56c0abda0 con 0x7fb56c0a98f0 2024-03-25T18:47:27.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.034+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3876291447 0 0) 0x7fb56c0abac0 con 0x7fb56c1430c0 2024-03-25T18:47:27.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.034+0000 7fb561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:47:27.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.034+0000 7fb561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:27.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.034+0000 7fb561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:27.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.034+0000 7fb561ffb640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:47:27.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.034+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb56c1380e0 con 0x7fb56c1430c0 2024-03-25T18:47:27.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.041+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1672292892 0 0) 0x7fb550004eb0 con 0x7fb56c13f710 2024-03-25T18:47:27.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.041+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb56c0abac0 con 0x7fb56c13f710 2024-03-25T18:47:27.042 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.041+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb55c001690 con 0x7fb56c13f710 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.041+0000 7fb561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.041+0000 7fb561ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.041+0000 7fb561ffb640 10 monclient(hunting): dump: 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:27.045 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:27.046 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:27.046 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.041+0000 7fb561ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.048+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2788139140 0 0) 0x7fb56c0abda0 con 0x7fb56c0a98f0 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.048+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb550004eb0 con 0x7fb56c0a98f0 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.048+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb5680018a0 con 0x7fb56c0a98f0 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.048+0000 7fb561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.048+0000 7fb561ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.048+0000 7fb561ffb640 10 monclient(hunting): dump: 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:27.049 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:27.050 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:27.050 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:27.050 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:27.050 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.053+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2322724644 0 0) 0x7fb56c1380e0 con 0x7fb56c1430c0 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.054+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb56c0abda0 con 0x7fb56c1430c0 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.054+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb564001bd0 con 0x7fb56c1430c0 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.054+0000 7fb561ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.054+0000 7fb561ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.054+0000 7fb561ffb640 10 monclient(hunting): dump: 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:27.055 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:27.056 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:27.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3543369722 0 0) 0x7fb56c0abda0 con 0x7fb56c1430c0 2024-03-25T18:47:27.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 >> v1:172.21.15.32:6790/0 conn(0x7fb56c0a98f0 legacy=0x7fb56c0ac580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:27.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 >> v1:172.21.15.136:6789/0 conn(0x7fb56c13f710 legacy=0x7fb56c138b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:27.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:47:27.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 1 monclient: found mon.a 2024-03-25T18:47:27.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:27.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:27.066 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56c137d90 con 0x7fb56c1430c0 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 10 monclient: _finish_auth 0 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb561ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb573893640 5 monclient: authenticate success, global_id 4605 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb573893640 10 monclient: _renew_subs 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb573893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56c139240 con 0x7fb56c1430c0 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb573893640 10 client.4605.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb573893640 10 monclient: _renew_subs 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb573893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.064+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56c136830 con 0x7fb56c1430c0 2024-03-25T18:47:27.067 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.065+0000 7fb573893640 1 librados: init done 2024-03-25T18:47:27.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.066+0000 7fb573893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:27.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.066+0000 7fb573893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:27.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.066+0000 7fb573893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:27.068 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.066+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb56c0627b0 con 0x7fb56c1430c0 2024-03-25T18:47:27.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.412+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb564001fe0 con 0x7fb56c1430c0 2024-03-25T18:47:27.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.412+0000 7fb561ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:27.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.412+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb564004d00 con 0x7fb56c1430c0 2024-03-25T18:47:27.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.412+0000 7fb561ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:27.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.412+0000 7fb561ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:27.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.412+0000 7fb561ffb640 10 monclient: dump: 2024-03-25T18:47:27.413 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:27.413 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:27.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.412+0000 7fb5637fe640 4 set_mon_vals no callback set 2024-03-25T18:47:27.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.413+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fb56401acd0 con 0x7fb56c1430c0 2024-03-25T18:47:27.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.413+0000 7fb57162f640 10 monclient: get_auth_request con 0x7fb55006e110 auth_method 0 2024-03-25T18:47:27.415 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.413+0000 7fb561ffb640 10 client.4605.objecter ms_handle_connect 0x7fb55006e110 2024-03-25T18:47:27.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.419+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fb5640549e0 con 0x7fb56c1430c0 2024-03-25T18:47:27.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.419+0000 7fb561ffb640 10 client.4605.objecter ms_dispatch 0x7fb56c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:47:27.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.419+0000 7fb561ffb640 3 client.4605.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:47:27.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.419+0000 7fb561ffb640 3 client.4605.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:47:27.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.419+0000 7fb561ffb640 20 client.4605.objecter dump_active .. 0 homeless 2024-03-25T18:47:27.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.419+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb564050320 con 0x7fb56c1430c0 2024-03-25T18:47:27.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.419+0000 7fb561ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:27.421 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.419+0000 7fb561ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:47:27.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.677+0000 7fb573893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:27.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.677+0000 7fb573893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:27.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.677+0000 7fb573893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:27.679 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.677+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb56c13f070 con 0x7fb56c1430c0 2024-03-25T18:47:27.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.732+0000 7fb561ffb640 1 -- 172.21.15.32:0/980220423 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fb564052420 con 0x7fb56c1430c0 2024-03-25T18:47:27.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.733+0000 7fb561ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:27.734 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.733+0000 7fb561ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:47:27.734 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:47:27.734 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:47:27.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 10 librados: watch_flush enter 2024-03-25T18:47:27.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 10 librados: watch_flush exit 2024-03-25T18:47:27.736 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 20 client.4605.objecter shutdown clearing up homeless session... 2024-03-25T18:47:27.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 10 client.4605.objecter successfully canceled tick 2024-03-25T18:47:27.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 >> v1:172.21.15.136:6800/1141502808 conn(0x7fb55006e110 legacy=0x7fb5500705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:27.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 10 monclient: shutdown 2024-03-25T18:47:27.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:27.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 >> v1:172.21.15.32:6789/0 conn(0x7fb56c1430c0 legacy=0x7fb56c1e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:27.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 shutdown_connections 2024-03-25T18:47:27.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.735+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 >> 172.21.15.32:0/980220423 conn(0x7fb56c0b7ea0 msgr2=0x7fb56c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:27.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.736+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 shutdown_connections 2024-03-25T18:47:27.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.736+0000 7fb573893640 1 -- 172.21.15.32:0/980220423 wait complete. 2024-03-25T18:47:27.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:27.736+0000 7fb573893640 1 librados: shutdown 2024-03-25T18:47:27.748 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:47:33.583 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:45.911 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:33.646 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:45.974 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:33.681 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:08:46.009 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:33.692 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:46.009 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:33.698 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:46.015 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:33.717 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:46.045 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:33.724 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:08:46.041 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:33.727 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:46.055 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:33.765 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:08:46.093 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:34.555 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.669+0000 7f1844d4d640 10 monclient: get_monmap_and_config 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.669+0000 7f1844d4d640 10 monclient: build_initial_monmap 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.669+0000 7f1844d4d640 10 monclient: monmap: 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:47:34.671360+0000 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:47:34.671360+0000 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:47:34.671 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.670+0000 7f1844d4d640 1 Processor -- start 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.670+0000 7f1844d4d640 1 -- start start 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.670+0000 7f1844d4d640 10 monclient: init 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f18400a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f184013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f18401430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1840142520 con 0x7f184013f710 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18400b28a0 con 0x7f18400a98f0 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18400b2a30 con 0x7f18401430c0 2024-03-25T18:47:34.673 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:34.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.671+0000 7f1844d4d640 10 monclient(hunting): authenticate will time out at 6560.861094s 2024-03-25T18:47:34.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.672+0000 7f183e59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f18400a98f0 0x7f18400a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:46642/0 (socket says 172.21.15.32:46642) 2024-03-25T18:47:34.674 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.672+0000 7f183e59c640 1 -- 172.21.15.32:0/218188059 learned_addr learned my addr 172.21.15.32:0/218188059 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:34.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.772+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410578462 0 0) 0x7f18400b28a0 con 0x7f18400a98f0 2024-03-25T18:47:34.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.772+0000 7f183d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:47:34.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.772+0000 7f183d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:34.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.772+0000 7f183d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:34.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.772+0000 7f183d59a640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:47:34.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.772+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1828003520 con 0x7f18400a98f0 2024-03-25T18:47:34.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.775+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328685037 0 0) 0x7f1840142520 con 0x7f184013f710 2024-03-25T18:47:34.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.775+0000 7f183d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:47:34.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.775+0000 7f183d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:34.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.775+0000 7f183d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:34.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.775+0000 7f183d59a640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:47:34.776 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.775+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18400b28a0 con 0x7f184013f710 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1560221068 0 0) 0x7f18400b28a0 con 0x7f184013f710 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 >> v1:172.21.15.32:6790/0 conn(0x7f18401430c0 legacy=0x7f184014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 >> v1:172.21.15.136:6789/0 conn(0x7f18400a98f0 legacy=0x7f18400a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 1 monclient: found mon.noname-a 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18400b44c0 con 0x7f184013f710 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 10 monclient: _finish_auth 0 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:47:34.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1834001670 con 0x7f184013f710 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f1844d4d640 5 monclient: authenticate success, global_id 4608 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f1844d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f183d59a640 10 monclient: dump: 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:34.797 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:34.798 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:34.798 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:34.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.794+0000 7f1844d4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:34.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.804+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18340017d0 con 0x7f184013f710 2024-03-25T18:47:34.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.804+0000 7f183d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:34.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.804+0000 7f183d59a640 1 -- 172.21.15.32:0/218188059 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1834001ba0 con 0x7f184013f710 2024-03-25T18:47:34.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.804+0000 7f183d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:34.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.804+0000 7f183d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:34.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.804+0000 7f183d59a640 10 monclient: dump: 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.804+0000 7f1844d4d640 10 monclient: get_monmap_and_config success 2024-03-25T18:47:34.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.804+0000 7f1844d4d640 4 set_mon_vals no callback set 2024-03-25T18:47:34.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.805+0000 7f1844d4d640 10 monclient: shutdown 2024-03-25T18:47:34.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.805+0000 7f1844d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:34.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.805+0000 7f1844d4d640 1 -- 172.21.15.32:0/218188059 >> v1:172.21.15.32:6789/0 conn(0x7f184013f710 legacy=0x7f18401429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:34.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.805+0000 7f1844d4d640 1 -- 172.21.15.32:0/218188059 shutdown_connections 2024-03-25T18:47:34.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.805+0000 7f1844d4d640 1 -- 172.21.15.32:0/218188059 >> 172.21.15.32:0/218188059 conn(0x7f18400b7ea0 msgr2=0x7f18400b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:34.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.805+0000 7f1844d4d640 1 -- 172.21.15.32:0/218188059 shutdown_connections 2024-03-25T18:47:34.807 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.805+0000 7f1844d4d640 1 -- 172.21.15.32:0/218188059 wait complete. 2024-03-25T18:47:34.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.806+0000 7f1844d4d640 10 monclient: build_initial_monmap 2024-03-25T18:47:34.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.806+0000 7f1844d4d640 10 monclient: monmap: 2024-03-25T18:47:34.809 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:34.809 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:47:34.809 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:47:34.809 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 1 librados: starting msgr at 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 1 librados: starting objecter 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 1 Processor -- start 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 1 -- start start 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 1 librados: setting wanted keys 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 1 librados: calling monclient init 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 10 monclient: init 2024-03-25T18:47:34.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.807+0000 7f1844d4d640 10 monclient(hunting): picked mon.noname-b con 0x7f18400a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f1844d4d640 10 monclient(hunting): picked mon.noname-c con 0x7f184013f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f1844d4d640 10 monclient(hunting): picked mon.noname-a con 0x7f18401430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f1844d4d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18401d7830 con 0x7f18401430c0 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f1844d4d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18401d7500 con 0x7f18400a98f0 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f1844d4d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1840137970 con 0x7f184013f710 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f1844d4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f1844d4d640 10 monclient(hunting): authenticate will time out at 6560.997603s 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f183e59c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f18400a98f0 0x7f1840138710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:46692/0 (socket says 172.21.15.32:46692) 2024-03-25T18:47:34.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f183e59c640 1 -- 172.21.15.32:0/1644794668 learned_addr learned my addr 172.21.15.32:0/1644794668 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:34.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f182effd640 10 client.?.objecter ms_handle_connect 0x7f18400a98f0 2024-03-25T18:47:34.812 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.808+0000 7f182effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:34.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.816+0000 7f182effd640 10 client.?.objecter ms_handle_connect 0x7f184013f710 2024-03-25T18:47:34.817 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.816+0000 7f182effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:34.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.870+0000 7f182effd640 10 client.?.objecter ms_handle_connect 0x7f18401430c0 2024-03-25T18:47:34.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.870+0000 7f182effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:34.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.885+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722606492 0 0) 0x7f18401d7830 con 0x7f18401430c0 2024-03-25T18:47:34.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.885+0000 7f182effd640 10 monclient(hunting): _negotiate 2024-03-25T18:47:34.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.885+0000 7f182effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:34.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.885+0000 7f182effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:34.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.885+0000 7f182effd640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:47:34.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.885+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1820004eb0 con 0x7f18401430c0 2024-03-25T18:47:34.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.900+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811901932 0 0) 0x7f1820004eb0 con 0x7f18401430c0 2024-03-25T18:47:34.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.900+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18401d7830 con 0x7f18401430c0 2024-03-25T18:47:34.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.900+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1838001b20 con 0x7f18401430c0 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.900+0000 7f182effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.900+0000 7f182effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.900+0000 7f182effd640 10 monclient(hunting): dump: 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:34.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.900+0000 7f182effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:47:34.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1237058404 0 0) 0x7f18401d7830 con 0x7f18401430c0 2024-03-25T18:47:34.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 >> v1:172.21.15.32:6790/0 conn(0x7f184013f710 legacy=0x7f18401d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:34.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 >> v1:172.21.15.136:6789/0 conn(0x7f18400a98f0 legacy=0x7f1840138710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:34.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 10 monclient: _finish_hunting 0 2024-03-25T18:47:34.916 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 1 monclient: found mon.a 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1840137b00 con 0x7f18401430c0 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 10 monclient: _finish_auth 0 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 10 monclient: _check_auth_tickets 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.914+0000 7f182effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 5 monclient: authenticate success, global_id 4611 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 10 monclient: _renew_subs 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1840135080 con 0x7f18401430c0 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 10 client.4611.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 10 monclient: _renew_subs 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18401e36d0 con 0x7f18401430c0 2024-03-25T18:47:34.917 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.915+0000 7f1844d4d640 1 librados: init done 2024-03-25T18:47:34.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.916+0000 7f1844d4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:34.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.916+0000 7f1844d4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:34.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.916+0000 7f1844d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:34.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:34.916+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18401d7690 con 0x7f18401430c0 2024-03-25T18:47:35.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.246+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1838002070 con 0x7f18401430c0 2024-03-25T18:47:35.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.246+0000 7f182effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:35.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.246+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f1838003970 con 0x7f18401430c0 2024-03-25T18:47:35.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.246+0000 7f182effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:35.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.246+0000 7f182effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:35.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.246+0000 7f182effd640 10 monclient: dump: 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:35.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.246+0000 7f183cd99640 4 set_mon_vals no callback set 2024-03-25T18:47:35.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.247+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f1838003b90 con 0x7f18401430c0 2024-03-25T18:47:35.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.247+0000 7f183e59c640 10 monclient: get_auth_request con 0x7f182006dd90 auth_method 0 2024-03-25T18:47:35.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.247+0000 7f182effd640 10 client.4611.objecter ms_handle_connect 0x7f182006dd90 2024-03-25T18:47:35.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.252+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f1838053670 con 0x7f18401430c0 2024-03-25T18:47:35.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.253+0000 7f182effd640 10 client.4611.objecter ms_dispatch 0x7f1840000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:47:35.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.253+0000 7f182effd640 3 client.4611.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:47:35.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.253+0000 7f182effd640 3 client.4611.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:47:35.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.253+0000 7f182effd640 20 client.4611.objecter dump_active .. 0 homeless 2024-03-25T18:47:35.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.253+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18380b08b0 con 0x7f18401430c0 2024-03-25T18:47:35.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.253+0000 7f182effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:35.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.253+0000 7f182effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:47:35.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.504+0000 7f1844d4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:35.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.504+0000 7f1844d4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:35.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.504+0000 7f1844d4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:35.505 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.504+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18400627b0 con 0x7f18401430c0 2024-03-25T18:47:35.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.530+0000 7f182effd640 1 -- 172.21.15.32:0/1644794668 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f183804efb0 con 0x7f18401430c0 2024-03-25T18:47:35.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.530+0000 7f182effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:35.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.530+0000 7f182effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:47:35.531 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:47:35.532 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 10 librados: watch_flush enter 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 10 librados: watch_flush exit 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 20 client.4611.objecter shutdown clearing up homeless session... 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 10 client.4611.objecter successfully canceled tick 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 >> v1:172.21.15.136:6800/1141502808 conn(0x7f182006dd90 legacy=0x7f1820070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 10 monclient: shutdown 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 >> v1:172.21.15.32:6789/0 conn(0x7f18401430c0 legacy=0x7f18401e1fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 shutdown_connections 2024-03-25T18:47:35.534 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 >> 172.21.15.32:0/1644794668 conn(0x7f18400b7ea0 msgr2=0x7f1840074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:35.535 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 shutdown_connections 2024-03-25T18:47:35.535 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 1 -- 172.21.15.32:0/1644794668 wait complete. 2024-03-25T18:47:35.535 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:35.532+0000 7f1844d4d640 1 librados: shutdown 2024-03-25T18:47:35.545 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:47:42.351 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.456+0000 7fdfcd70e640 10 monclient: get_monmap_and_config 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.456+0000 7fdfcd70e640 10 monclient: build_initial_monmap 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.456+0000 7fdfcd70e640 10 monclient: monmap: 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:47:42.457642+0000 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:47:42.457642+0000 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:42.457 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:42.458 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:47:42.458 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:47:42.458 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 1 Processor -- start 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 1 -- start start 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 10 monclient: init 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc813fc60 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc8070570 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc80727a0 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:42.459 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc813c9f0 con 0x7fdfc813fc60 2024-03-25T18:47:42.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc806fac0 con 0x7fdfc80727a0 2024-03-25T18:47:42.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc80715d0 con 0x7fdfc8070570 2024-03-25T18:47:42.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:42.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.457+0000 7fdfcd70e640 10 monclient(hunting): authenticate will time out at 6568.647401s 2024-03-25T18:47:42.464 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.463+0000 7fdfc6ffd640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fdfc813fc60 0x7fdfc806f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:53978/0 (socket says 172.21.15.32:53978) 2024-03-25T18:47:42.464 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.463+0000 7fdfc6ffd640 1 -- 172.21.15.32:0/3131482268 learned_addr learned my addr 172.21.15.32:0/3131482268 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:42.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.485+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653249817 0 0) 0x7fdfc80715d0 con 0x7fdfc8070570 2024-03-25T18:47:42.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.485+0000 7fdfc67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:42.486 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.485+0000 7fdfc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:42.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.485+0000 7fdfc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:42.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.485+0000 7fdfc67fc640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:47:42.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.485+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfb4003520 con 0x7fdfc8070570 2024-03-25T18:47:42.491 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.489+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2397739807 0 0) 0x7fdfc813c9f0 con 0x7fdfc813fc60 2024-03-25T18:47:42.491 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.490+0000 7fdfc67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:42.491 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.490+0000 7fdfc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:42.491 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.490+0000 7fdfc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:42.491 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.490+0000 7fdfc67fc640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:47:42.491 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.490+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc80715d0 con 0x7fdfc813fc60 2024-03-25T18:47:42.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.492+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1556526819 0 0) 0x7fdfc806fac0 con 0x7fdfc80727a0 2024-03-25T18:47:42.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.492+0000 7fdfc67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:42.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.492+0000 7fdfc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:42.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.492+0000 7fdfc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:42.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.492+0000 7fdfc67fc640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:47:42.493 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.492+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc813c9f0 con 0x7fdfc80727a0 2024-03-25T18:47:42.502 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1375231381 0 0) 0x7fdfb4003520 con 0x7fdfc8070570 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 >> v1:172.21.15.136:6789/0 conn(0x7fdfc80727a0 legacy=0x7fdfc814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 >> v1:172.21.15.32:6789/0 conn(0x7fdfc813fc60 legacy=0x7fdfc806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 1 monclient: found mon.noname-b 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc80718b0 con 0x7fdfc8070570 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 10 monclient: _finish_auth 0 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fdfb0001690 con 0x7fdfc8070570 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfc67fc640 10 monclient: dump: 2024-03-25T18:47:42.503 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfcd70e640 5 monclient: authenticate success, global_id 4589 2024-03-25T18:47:42.504 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.501+0000 7fdfcd70e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfb00017f0 con 0x7fdfc8070570 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfc67fc640 1 -- 172.21.15.32:0/3131482268 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fdfb0001c00 con 0x7fdfc8070570 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfcd70e640 10 monclient: get_monmap_and_config success 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfcd70e640 4 set_mon_vals no callback set 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfc67fc640 10 monclient: dump: 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.521 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:42.522 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:42.522 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:42.522 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:42.522 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:42.522 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:42.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfcd70e640 10 monclient: shutdown 2024-03-25T18:47:42.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfcd70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:42.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfcd70e640 1 -- 172.21.15.32:0/3131482268 >> v1:172.21.15.32:6790/0 conn(0x7fdfc8070570 legacy=0x7fdfc8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:42.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfcd70e640 1 -- 172.21.15.32:0/3131482268 shutdown_connections 2024-03-25T18:47:42.523 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.520+0000 7fdfcd70e640 1 -- 172.21.15.32:0/3131482268 >> 172.21.15.32:0/3131482268 conn(0x7fdfc80b4a80 msgr2=0x7fdfc80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.521+0000 7fdfcd70e640 1 -- 172.21.15.32:0/3131482268 shutdown_connections 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.521+0000 7fdfcd70e640 1 -- 172.21.15.32:0/3131482268 wait complete. 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.521+0000 7fdfcd70e640 10 monclient: build_initial_monmap 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.521+0000 7fdfcd70e640 10 monclient: monmap: 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:42.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.522+0000 7fdfcd70e640 1 librados: starting msgr at 2024-03-25T18:47:42.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.522+0000 7fdfcd70e640 1 librados: starting objecter 2024-03-25T18:47:42.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.522+0000 7fdfcd70e640 1 Processor -- start 2024-03-25T18:47:42.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.522+0000 7fdfcd70e640 1 -- start start 2024-03-25T18:47:42.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.522+0000 7fdfcd70e640 1 librados: setting wanted keys 2024-03-25T18:47:42.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.522+0000 7fdfcd70e640 1 librados: calling monclient init 2024-03-25T18:47:42.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.522+0000 7fdfcd70e640 10 monclient: init 2024-03-25T18:47:42.525 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 10 monclient(hunting): picked mon.noname-b con 0x7fdfc8070570 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 10 monclient(hunting): picked mon.noname-c con 0x7fdfc80727a0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 10 monclient(hunting): picked mon.noname-a con 0x7fdfc813fc60 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc81399c0 con 0x7fdfc813fc60 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc8139530 con 0x7fdfc8070570 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdfc8140610 con 0x7fdfc80727a0 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:42.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.523+0000 7fdfcd70e640 10 monclient(hunting): authenticate will time out at 6568.712707s 2024-03-25T18:47:42.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.530+0000 7fdfc6ffd640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fdfc8070570 0x7fdfc81dad50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:38808/0 (socket says 172.21.15.32:38808) 2024-03-25T18:47:42.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.530+0000 7fdfc6ffd640 1 -- 172.21.15.32:0/1703722096 learned_addr learned my addr 172.21.15.32:0/1703722096 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:42.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.532+0000 7fdfbf7fe640 10 client.?.objecter ms_handle_connect 0x7fdfc8070570 2024-03-25T18:47:42.533 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.532+0000 7fdfbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:42.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.537+0000 7fdfbf7fe640 10 client.?.objecter ms_handle_connect 0x7fdfc813fc60 2024-03-25T18:47:42.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.537+0000 7fdfbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:42.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.542+0000 7fdfbf7fe640 10 client.?.objecter ms_handle_connect 0x7fdfc80727a0 2024-03-25T18:47:42.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.542+0000 7fdfbf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:42.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.544+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2724692312 0 0) 0x7fdfc813c9f0 con 0x7fdfc813fc60 2024-03-25T18:47:42.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.544+0000 7fdfbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:47:42.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.544+0000 7fdfbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:42.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.544+0000 7fdfbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:42.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.544+0000 7fdfbf7fe640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:47:42.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.544+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfa8004eb0 con 0x7fdfc813fc60 2024-03-25T18:47:42.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.545+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2340869340 0 0) 0x7fdfc8139530 con 0x7fdfc8070570 2024-03-25T18:47:42.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.545+0000 7fdfbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:47:42.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.545+0000 7fdfbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:42.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.545+0000 7fdfbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:42.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.545+0000 7fdfbf7fe640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:47:42.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.545+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc813c9f0 con 0x7fdfc8070570 2024-03-25T18:47:42.560 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.559+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1892025978 0 0) 0x7fdfc8140610 con 0x7fdfc80727a0 2024-03-25T18:47:42.560 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.559+0000 7fdfbf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:47:42.560 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.559+0000 7fdfbf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:42.560 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.559+0000 7fdfbf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:42.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.559+0000 7fdfbf7fe640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:47:42.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.559+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdfc8139530 con 0x7fdfc80727a0 2024-03-25T18:47:42.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.560+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2168625755 0 0) 0x7fdfc813c9f0 con 0x7fdfc8070570 2024-03-25T18:47:42.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.560+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfc8140610 con 0x7fdfc8070570 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.560+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fdfb80018d0 con 0x7fdfc8070570 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.560+0000 7fdfbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.560+0000 7fdfbf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.560+0000 7fdfbf7fe640 10 monclient(hunting): dump: 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:42.562 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:42.563 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.560+0000 7fdfbf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:47:42.563 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.561+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 381316472 0 0) 0x7fdfa8004eb0 con 0x7fdfc813fc60 2024-03-25T18:47:42.563 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.561+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdfc813c9f0 con 0x7fdfc813fc60 2024-03-25T18:47:42.563 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.561+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fdfc0001ad0 con 0x7fdfc813fc60 2024-03-25T18:47:42.563 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.561+0000 7fdfbf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.561+0000 7fdfbf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.561+0000 7fdfbf7fe640 10 monclient(hunting): dump: 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:42.564 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1217033531 0 0) 0x7fdfc813c9f0 con 0x7fdfc813fc60 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 >> v1:172.21.15.32:6790/0 conn(0x7fdfc80727a0 legacy=0x7fdfc81329c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 >> v1:172.21.15.136:6789/0 conn(0x7fdfc8070570 legacy=0x7fdfc81dad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 1 monclient: found mon.a 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfc81337e0 con 0x7fdfc813fc60 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:47:42.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfbf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 10 monclient: _renew_subs 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfc8138f90 con 0x7fdfc813fc60 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 10 client.4617.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 10 monclient: _renew_subs 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfc8138bc0 con 0x7fdfc813fc60 2024-03-25T18:47:42.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.569+0000 7fdfcd70e640 1 librados: init done 2024-03-25T18:47:42.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.570+0000 7fdfcd70e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:42.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.570+0000 7fdfcd70e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:42.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.570+0000 7fdfcd70e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:42.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.570+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfc81392e0 con 0x7fdfc813fc60 2024-03-25T18:47:42.884 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.883+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdfc0001f60 con 0x7fdfc813fc60 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.883+0000 7fdfbf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.883+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fdfc0003b30 con 0x7fdfc813fc60 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.883+0000 7fdfbf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.883+0000 7fdfbf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.883+0000 7fdfbf7fe640 10 monclient: dump: 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:42.885 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:42.886 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:42.886 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:42.886 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:42.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.883+0000 7fdfc5ffb640 4 set_mon_vals no callback set 2024-03-25T18:47:42.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.884+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fdfc0019b00 con 0x7fdfc813fc60 2024-03-25T18:47:42.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.885+0000 7fdfc6ffd640 10 monclient: get_auth_request con 0x7fdfa806dfb0 auth_method 0 2024-03-25T18:47:42.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.885+0000 7fdfbf7fe640 10 client.4617.objecter ms_handle_connect 0x7fdfa806dfb0 2024-03-25T18:47:42.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.890+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fdfc0053740 con 0x7fdfc813fc60 2024-03-25T18:47:42.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.890+0000 7fdfbf7fe640 10 client.4617.objecter ms_dispatch 0x7fdfc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:47:42.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.890+0000 7fdfbf7fe640 3 client.4617.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:47:42.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.890+0000 7fdfbf7fe640 3 client.4617.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:47:42.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.890+0000 7fdfbf7fe640 20 client.4617.objecter dump_active .. 0 homeless 2024-03-25T18:47:42.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.890+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdfc004f080 con 0x7fdfc813fc60 2024-03-25T18:47:42.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.890+0000 7fdfbf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:42.891 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:42.890+0000 7fdfbf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:47:43.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.156+0000 7fdfcd70e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:43.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.156+0000 7fdfcd70e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:43.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.156+0000 7fdfcd70e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:47:43.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.156+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdfc80627b0 con 0x7fdfc813fc60 2024-03-25T18:47:43.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.178+0000 7fdfbf7fe640 1 -- 172.21.15.32:0/1703722096 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fdfc0051180 con 0x7fdfc813fc60 2024-03-25T18:47:43.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.178+0000 7fdfbf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:43.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.178+0000 7fdfbf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:47:43.180 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:47:43.180 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:47:43.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.180+0000 7fdfcd70e640 10 librados: watch_flush enter 2024-03-25T18:47:43.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.180+0000 7fdfcd70e640 10 librados: watch_flush exit 2024-03-25T18:47:43.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.180+0000 7fdfcd70e640 20 client.4617.objecter shutdown clearing up homeless session... 2024-03-25T18:47:43.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.180+0000 7fdfcd70e640 10 client.4617.objecter successfully canceled tick 2024-03-25T18:47:43.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 >> v1:172.21.15.136:6800/1141502808 conn(0x7fdfa806dfb0 legacy=0x7fdfa8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:43.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 10 monclient: shutdown 2024-03-25T18:47:43.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:43.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 >> v1:172.21.15.32:6789/0 conn(0x7fdfc813fc60 legacy=0x7fdfc81330d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:43.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 shutdown_connections 2024-03-25T18:47:43.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 >> 172.21.15.32:0/1703722096 conn(0x7fdfc80b4a80 msgr2=0x7fdfc80b4e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:43.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 shutdown_connections 2024-03-25T18:47:43.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 1 -- 172.21.15.32:0/1703722096 wait complete. 2024-03-25T18:47:43.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:43.181+0000 7fdfcd70e640 1 librados: shutdown 2024-03-25T18:47:43.194 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:47:50.001 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.118+0000 7fb87d18d640 10 monclient: get_monmap_and_config 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.118+0000 7fb87d18d640 10 monclient: build_initial_monmap 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.118+0000 7fb87d18d640 10 monclient: monmap: 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:47:50.120366+0000 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:47:50.120366+0000 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:47:50.120 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:50.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.119+0000 7fb87d18d640 1 Processor -- start 2024-03-25T18:47:50.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.119+0000 7fb87d18d640 1 -- start start 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.119+0000 7fb87d18d640 10 monclient: init 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 10 monclient(hunting): picked mon.noname-b con 0x7fb8780a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 10 monclient(hunting): picked mon.noname-a con 0x7fb87813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 10 monclient(hunting): picked mon.noname-c con 0x7fb8781430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878142520 con 0x7fb87813f710 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780b28a0 con 0x7fb8781430c0 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8780b2a30 con 0x7fb8780a98f0 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:50.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.120+0000 7fb87d18d640 10 monclient(hunting): authenticate will time out at 6576.310098s 2024-03-25T18:47:50.135 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.134+0000 7fb87759e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb8781430c0 0x7fb87814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:37022/0 (socket says 172.21.15.32:37022) 2024-03-25T18:47:50.136 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.134+0000 7fb87759e640 1 -- 172.21.15.32:0/3601209122 learned_addr learned my addr 172.21.15.32:0/3601209122 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:50.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.156+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2759424628 0 0) 0x7fb8780b28a0 con 0x7fb8781430c0 2024-03-25T18:47:50.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.156+0000 7fb87659c640 10 monclient(hunting): _negotiate 2024-03-25T18:47:50.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.156+0000 7fb87659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:50.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.156+0000 7fb87659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:50.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.156+0000 7fb87659c640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:47:50.157 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.156+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb864003520 con 0x7fb8781430c0 2024-03-25T18:47:50.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.158+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4055474418 0 0) 0x7fb8780b2a30 con 0x7fb8780a98f0 2024-03-25T18:47:50.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.158+0000 7fb87659c640 10 monclient(hunting): _negotiate 2024-03-25T18:47:50.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.158+0000 7fb87659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:50.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.158+0000 7fb87659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:50.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.158+0000 7fb87659c640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:47:50.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.158+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8780b28a0 con 0x7fb8780a98f0 2024-03-25T18:47:50.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680106795 0 0) 0x7fb878142520 con 0x7fb87813f710 2024-03-25T18:47:50.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient(hunting): _negotiate 2024-03-25T18:47:50.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:50.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:50.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:47:50.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8780b2a30 con 0x7fb87813f710 2024-03-25T18:47:50.173 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1453265824 0 0) 0x7fb864003520 con 0x7fb8781430c0 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 >> v1:172.21.15.32:6790/0 conn(0x7fb8780a98f0 legacy=0x7fb8780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 >> v1:172.21.15.32:6789/0 conn(0x7fb87813f710 legacy=0x7fb8781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient: _finish_hunting 0 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 1 monclient: found mon.noname-c 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8780b44c0 con 0x7fb8781430c0 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient: _finish_auth 0 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient: _check_auth_tickets 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb870001a10 con 0x7fb8781430c0 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:47:50.174 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87659c640 10 monclient: dump: 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87d18d640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:47:50.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.171+0000 7fb87d18d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:50.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.180+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb870001b70 con 0x7fb8781430c0 2024-03-25T18:47:50.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.180+0000 7fb87659c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:50.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.180+0000 7fb87659c640 1 -- 172.21.15.32:0/3601209122 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb870001f80 con 0x7fb8781430c0 2024-03-25T18:47:50.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.180+0000 7fb87659c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:50.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.180+0000 7fb87659c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.180+0000 7fb87659c640 10 monclient: dump: 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.180+0000 7fb87d18d640 10 monclient: get_monmap_and_config success 2024-03-25T18:47:50.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.180+0000 7fb87d18d640 4 set_mon_vals no callback set 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.181+0000 7fb87d18d640 10 monclient: shutdown 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.181+0000 7fb87d18d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.181+0000 7fb87d18d640 1 -- 172.21.15.32:0/3601209122 >> v1:172.21.15.136:6789/0 conn(0x7fb8781430c0 legacy=0x7fb87814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.181+0000 7fb87d18d640 1 -- 172.21.15.32:0/3601209122 shutdown_connections 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.181+0000 7fb87d18d640 1 -- 172.21.15.32:0/3601209122 >> 172.21.15.32:0/3601209122 conn(0x7fb8780b7ea0 msgr2=0x7fb8780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.181+0000 7fb87d18d640 1 -- 172.21.15.32:0/3601209122 shutdown_connections 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.181+0000 7fb87d18d640 1 -- 172.21.15.32:0/3601209122 wait complete. 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.182+0000 7fb87d18d640 10 monclient: build_initial_monmap 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.182+0000 7fb87d18d640 10 monclient: monmap: 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:50.184 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.182+0000 7fb87d18d640 1 librados: starting msgr at 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.182+0000 7fb87d18d640 1 librados: starting objecter 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 1 Processor -- start 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 1 -- start start 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 1 librados: setting wanted keys 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 1 librados: calling monclient init 2024-03-25T18:47:50.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 10 monclient: init 2024-03-25T18:47:50.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:50.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:47:50.186 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 10 monclient(hunting): picked mon.noname-a con 0x7fb8780a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:50.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 10 monclient(hunting): picked mon.noname-c con 0x7fb87813f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:50.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 10 monclient(hunting): picked mon.noname-b con 0x7fb8781430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:50.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878135230 con 0x7fb8780a98f0 2024-03-25T18:47:50.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878135da0 con 0x7fb8781430c0 2024-03-25T18:47:50.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb878135f30 con 0x7fb87813f710 2024-03-25T18:47:50.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:50.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.183+0000 7fb87d18d640 10 monclient(hunting): authenticate will time out at 6576.373416s 2024-03-25T18:47:50.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.192+0000 7fb87759e640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fb8781430c0 0x7fb8781db560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:37070/0 (socket says 172.21.15.32:37070) 2024-03-25T18:47:50.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.192+0000 7fb87759e640 1 -- 172.21.15.32:0/3406817221 learned_addr learned my addr 172.21.15.32:0/3406817221 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:50.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.194+0000 7fb86f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8781430c0 2024-03-25T18:47:50.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.194+0000 7fb86f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:50.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.200+0000 7fb86f7fe640 10 client.?.objecter ms_handle_connect 0x7fb87813f710 2024-03-25T18:47:50.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.200+0000 7fb86f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:50.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.201+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036069771 0 0) 0x7fb878135da0 con 0x7fb8781430c0 2024-03-25T18:47:50.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.201+0000 7fb86f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:47:50.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.201+0000 7fb86f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:50.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.201+0000 7fb86f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:50.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.201+0000 7fb86f7fe640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:47:50.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.201+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858004eb0 con 0x7fb8781430c0 2024-03-25T18:47:50.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.206+0000 7fb86f7fe640 10 client.?.objecter ms_handle_connect 0x7fb8780a98f0 2024-03-25T18:47:50.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.206+0000 7fb86f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:50.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.219+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3718445220 0 0) 0x7fb858004eb0 con 0x7fb8781430c0 2024-03-25T18:47:50.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.219+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb878135da0 con 0x7fb8781430c0 2024-03-25T18:47:50.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.219+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb870001ec0 con 0x7fb8781430c0 2024-03-25T18:47:50.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.219+0000 7fb86f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:50.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.219+0000 7fb86f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:47:50.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.219+0000 7fb86f7fe640 10 monclient(hunting): dump: 2024-03-25T18:47:50.220 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:50.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.219+0000 7fb86f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:47:50.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.224+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251126565 0 0) 0x7fb878135f30 con 0x7fb87813f710 2024-03-25T18:47:50.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.224+0000 7fb86f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:47:50.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.224+0000 7fb86f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:50.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.224+0000 7fb86f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:50.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.224+0000 7fb86f7fe640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:47:50.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.224+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb858004eb0 con 0x7fb87813f710 2024-03-25T18:47:50.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 913839788 0 0) 0x7fb878135da0 con 0x7fb8781430c0 2024-03-25T18:47:50.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 >> v1:172.21.15.32:6790/0 conn(0x7fb87813f710 legacy=0x7fb8781dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:50.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 >> v1:172.21.15.32:6789/0 conn(0x7fb8780a98f0 legacy=0x7fb8781d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:50.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:47:50.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 1 monclient: found mon.b 2024-03-25T18:47:50.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8781360c0 con 0x7fb8781430c0 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.237+0000 7fb86f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 5 monclient: authenticate success, global_id 4600 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 10 monclient: _renew_subs 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8781d5000 con 0x7fb8781430c0 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 10 client.4600.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 10 monclient: _renew_subs 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8781d5350 con 0x7fb8781430c0 2024-03-25T18:47:50.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.238+0000 7fb87d18d640 1 librados: init done 2024-03-25T18:47:50.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.239+0000 7fb87d18d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:50.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.239+0000 7fb87d18d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:50.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.239+0000 7fb87d18d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:50.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.239+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8781d4530 con 0x7fb8781430c0 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.509+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb870004e10 con 0x7fb8781430c0 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.509+0000 7fb86f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.509+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fb870005120 con 0x7fb8781430c0 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.509+0000 7fb86f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.509+0000 7fb86f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.509+0000 7fb86f7fe640 10 monclient: dump: 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:50.511 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:50.512 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:50.512 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:50.512 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:50.512 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:50.512 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:50.512 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:50.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.509+0000 7fb875d9b640 4 set_mon_vals no callback set 2024-03-25T18:47:50.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.510+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fb87001b0f0 con 0x7fb8781430c0 2024-03-25T18:47:50.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.511+0000 7fb876d9d640 10 monclient: get_auth_request con 0x7fb85806dd90 auth_method 0 2024-03-25T18:47:50.513 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.511+0000 7fb86f7fe640 10 client.4600.objecter ms_handle_connect 0x7fb85806dd90 2024-03-25T18:47:50.516 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.515+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fb870054b60 con 0x7fb8781430c0 2024-03-25T18:47:50.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.515+0000 7fb86f7fe640 10 client.4600.objecter ms_dispatch 0x7fb878000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:47:50.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.515+0000 7fb86f7fe640 3 client.4600.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:47:50.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.515+0000 7fb86f7fe640 3 client.4600.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:47:50.517 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.516+0000 7fb86f7fe640 20 client.4600.objecter dump_active .. 0 homeless 2024-03-25T18:47:50.518 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.517+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb870050670 con 0x7fb8781430c0 2024-03-25T18:47:50.518 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.517+0000 7fb86f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:50.518 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.517+0000 7fb86f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:47:50.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.727+0000 7fb87d18d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:50.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.727+0000 7fb87d18d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:50.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.727+0000 7fb87d18d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:50.729 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.727+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb878135090 con 0x7fb8781430c0 2024-03-25T18:47:50.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.794+0000 7fb86f7fe640 1 -- 172.21.15.32:0/3406817221 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fb878135090 con 0x7fb8781430c0 2024-03-25T18:47:50.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.794+0000 7fb86f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:50.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.794+0000 7fb86f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:47:50.796 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:47:50.796 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:47:50.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.796+0000 7fb87d18d640 10 librados: watch_flush enter 2024-03-25T18:47:50.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.796+0000 7fb87d18d640 10 librados: watch_flush exit 2024-03-25T18:47:50.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.796+0000 7fb87d18d640 20 client.4600.objecter shutdown clearing up homeless session... 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.796+0000 7fb87d18d640 10 client.4600.objecter successfully canceled tick 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 >> v1:172.21.15.136:6800/1141502808 conn(0x7fb85806dd90 legacy=0x7fb858070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 10 monclient: shutdown 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 >> v1:172.21.15.136:6789/0 conn(0x7fb8781430c0 legacy=0x7fb8781db560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 shutdown_connections 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 >> 172.21.15.32:0/3406817221 conn(0x7fb8780b7ea0 msgr2=0x7fb87814ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 shutdown_connections 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 1 -- 172.21.15.32:0/3406817221 wait complete. 2024-03-25T18:47:50.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:50.797+0000 7fb87d18d640 1 librados: shutdown 2024-03-25T18:47:50.811 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:47:53.765 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:06.092 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:53.774 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:06.102 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:53.811 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:06.128 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:53.813 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:06.141 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:53.817 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:06.134 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:53.821 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:06.149 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:53.838 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:06.166 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:53.848 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:06.165 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:53.871 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:06.199 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:47:57.618 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:47:57.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.737+0000 7fac1bfcd640 10 monclient: get_monmap_and_config 2024-03-25T18:47:57.737 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.737+0000 7fac1bfcd640 10 monclient: build_initial_monmap 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.737+0000 7fac1bfcd640 10 monclient: monmap: 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:47:57.737776+0000 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:47:57.737776+0000 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:57.738 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.737+0000 7fac1bfcd640 1 Processor -- start 2024-03-25T18:47:57.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.738+0000 7fac1bfcd640 1 -- start start 2024-03-25T18:47:57.739 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.738+0000 7fac1bfcd640 10 monclient: init 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.738+0000 7fac1bfcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.738+0000 7fac1bfcd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.738+0000 7fac1bfcd640 10 monclient(hunting): picked mon.noname-b con 0x7fac140a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.738+0000 7fac1bfcd640 10 monclient(hunting): picked mon.noname-a con 0x7fac1413f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.739+0000 7fac1bfcd640 10 monclient(hunting): picked mon.noname-c con 0x7fac141430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.739+0000 7fac1bfcd640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac14142520 con 0x7fac1413f710 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.739+0000 7fac1bfcd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac140b28a0 con 0x7fac141430c0 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.739+0000 7fac1bfcd640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac140b2a30 con 0x7fac140a98f0 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.739+0000 7fac1bfcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:57.740 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.739+0000 7fac1bfcd640 10 monclient(hunting): authenticate will time out at 6583.927685s 2024-03-25T18:47:57.750 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.750+0000 7fac19d69640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fac140a98f0 0x7fac140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:48138/0 (socket says 172.21.15.32:48138) 2024-03-25T18:47:57.750 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.750+0000 7fac19d69640 1 -- 172.21.15.32:0/2554695004 learned_addr learned my addr 172.21.15.32:0/2554695004 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:57.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.760+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211310806 0 0) 0x7fac140b28a0 con 0x7fac141430c0 2024-03-25T18:47:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.760+0000 7fac18d67640 10 monclient(hunting): _negotiate 2024-03-25T18:47:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.760+0000 7fac18d67640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.760+0000 7fac18d67640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.760+0000 7fac18d67640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:47:57.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.760+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac00003520 con 0x7fac141430c0 2024-03-25T18:47:57.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 96355917 0 0) 0x7fac00003520 con 0x7fac141430c0 2024-03-25T18:47:57.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 >> v1:172.21.15.32:6790/0 conn(0x7fac140a98f0 legacy=0x7fac140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:57.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 >> v1:172.21.15.32:6789/0 conn(0x7fac1413f710 legacy=0x7fac141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:57.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 10 monclient: _finish_hunting 0 2024-03-25T18:47:57.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 1 monclient: found mon.noname-c 2024-03-25T18:47:57.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:57.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:47:57.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac140b44c0 con 0x7fac141430c0 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 10 monclient: _finish_auth 0 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 10 monclient: _check_auth_tickets 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fac0c001a10 con 0x7fac141430c0 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac18d67640 10 monclient: dump: 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:57.773 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:57.774 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:57.774 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:57.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac1bfcd640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:47:57.774 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.772+0000 7fac1bfcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:47:57.791 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.791+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac0c001b70 con 0x7fac141430c0 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.791+0000 7fac18d67640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.791+0000 7fac18d67640 1 -- 172.21.15.32:0/2554695004 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fac0c001fc0 con 0x7fac141430c0 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.791+0000 7fac1bfcd640 10 monclient: get_monmap_and_config success 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.791+0000 7fac1bfcd640 4 set_mon_vals no callback set 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.791+0000 7fac18d67640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.791+0000 7fac18d67640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.791+0000 7fac18d67640 10 monclient: dump: 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:57.792 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:57.793 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:57.793 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:57.793 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.792+0000 7fac1bfcd640 10 monclient: shutdown 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.792+0000 7fac1bfcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.792+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2554695004 >> v1:172.21.15.136:6789/0 conn(0x7fac141430c0 legacy=0x7fac1414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.792+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2554695004 shutdown_connections 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.792+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2554695004 >> 172.21.15.32:0/2554695004 conn(0x7fac140b7ea0 msgr2=0x7fac140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.792+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2554695004 shutdown_connections 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.792+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2554695004 wait complete. 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.793+0000 7fac1bfcd640 10 monclient: build_initial_monmap 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.793+0000 7fac1bfcd640 10 monclient: monmap: 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:47:57.795 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.793+0000 7fac1bfcd640 1 librados: starting msgr at 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.793+0000 7fac1bfcd640 1 librados: starting objecter 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 1 Processor -- start 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 1 -- start start 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 1 librados: setting wanted keys 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 1 librados: calling monclient init 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 10 monclient: init 2024-03-25T18:47:57.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 10 monclient(hunting): picked mon.noname-a con 0x7fac140a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 10 monclient(hunting): picked mon.noname-b con 0x7fac1413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 10 monclient(hunting): picked mon.noname-c con 0x7fac141430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac14137ab0 con 0x7fac140a98f0 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac14138bd0 con 0x7fac1413f710 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac14138e40 con 0x7fac141430c0 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:47:57.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.794+0000 7fac1bfcd640 10 monclient(hunting): authenticate will time out at 6583.983300s 2024-03-25T18:47:57.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.799+0000 7fac1a56a640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fac141430c0 0x7fac141e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:48190/0 (socket says 172.21.15.32:48190) 2024-03-25T18:47:57.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.799+0000 7fac1a56a640 1 -- 172.21.15.32:0/2879014794 learned_addr learned my addr 172.21.15.32:0/2879014794 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:47:57.801 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.801+0000 7fac0a7fc640 10 client.?.objecter ms_handle_connect 0x7fac141430c0 2024-03-25T18:47:57.801 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.801+0000 7fac0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:57.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.809+0000 7fac0a7fc640 10 client.?.objecter ms_handle_connect 0x7fac1413f710 2024-03-25T18:47:57.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.809+0000 7fac0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:57.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.821+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4051725148 0 0) 0x7fac14138bd0 con 0x7fac1413f710 2024-03-25T18:47:57.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.821+0000 7fac0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:57.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.821+0000 7fac0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:57.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.821+0000 7fac0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:57.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.821+0000 7fac0a7fc640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:47:57.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.821+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf8004eb0 con 0x7fac1413f710 2024-03-25T18:47:57.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.824+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2553649522 0 0) 0x7fac14138e40 con 0x7fac141430c0 2024-03-25T18:47:57.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.824+0000 7fac0a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:47:57.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.824+0000 7fac0a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:47:57.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.824+0000 7fac0a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:47:57.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.824+0000 7fac0a7fc640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:47:57.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.824+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fac14138bd0 con 0x7fac141430c0 2024-03-25T18:47:57.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.824+0000 7fac0a7fc640 10 client.?.objecter ms_handle_connect 0x7fac140a98f0 2024-03-25T18:47:57.824 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.824+0000 7fac0a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:47:57.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.830+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 43190249 0 0) 0x7fabf8004eb0 con 0x7fac1413f710 2024-03-25T18:47:57.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.830+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac14138e40 con 0x7fac1413f710 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.830+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fac04001930 con 0x7fac1413f710 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.830+0000 7fac0a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.830+0000 7fac0a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.830+0000 7fac0a7fc640 10 monclient(hunting): dump: 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:57.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.830+0000 7fac0a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3769331970 0 0) 0x7fac14138e40 con 0x7fac1413f710 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 >> v1:172.21.15.32:6790/0 conn(0x7fac141430c0 legacy=0x7fac141e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 >> v1:172.21.15.32:6789/0 conn(0x7fac140a98f0 legacy=0x7fac14137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 1 monclient: found mon.b 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac14139120 con 0x7fac1413f710 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac0a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:47:57.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:47:57.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 10 monclient: _renew_subs 2024-03-25T18:47:57.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:57.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac14139400 con 0x7fac1413f710 2024-03-25T18:47:57.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 10 client.4606.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:47:57.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 10 monclient: _renew_subs 2024-03-25T18:47:57.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:57.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac140b7880 con 0x7fac1413f710 2024-03-25T18:47:57.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.838+0000 7fac1bfcd640 1 librados: init done 2024-03-25T18:47:57.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.840+0000 7fac1bfcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:57.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.840+0000 7fac1bfcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:57.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.840+0000 7fac1bfcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:57.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.840+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac14137810 con 0x7fac1413f710 2024-03-25T18:47:57.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.862+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac04003610 con 0x7fac1413f710 2024-03-25T18:47:57.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.862+0000 7fac0a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:47:57.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.862+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fac04003a20 con 0x7fac1413f710 2024-03-25T18:47:57.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.862+0000 7fac0a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:47:57.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.862+0000 7fac0a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:47:57.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.862+0000 7fac0a7fc640 10 monclient: dump: 2024-03-25T18:47:57.862 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:47:57.862 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:47:57.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.862+0000 7fac0bfff640 4 set_mon_vals no callback set 2024-03-25T18:47:57.864 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.863+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fac04003c40 con 0x7fac1413f710 2024-03-25T18:47:57.864 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.863+0000 7fac19d69640 10 monclient: get_auth_request con 0x7fabf806dd10 auth_method 0 2024-03-25T18:47:57.864 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.864+0000 7fac0a7fc640 10 client.4606.objecter ms_handle_connect 0x7fabf806dd10 2024-03-25T18:47:57.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.867+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7fac04053610 con 0x7fac1413f710 2024-03-25T18:47:57.867 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.867+0000 7fac0a7fc640 10 client.4606.objecter ms_dispatch 0x7fac14000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:47:57.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.867+0000 7fac0a7fc640 3 client.4606.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:47:57.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.867+0000 7fac0a7fc640 3 client.4606.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:47:57.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:57.867+0000 7fac0a7fc640 20 client.4606.objecter dump_active .. 0 homeless 2024-03-25T18:47:58.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.127+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac0404f120 con 0x7fac1413f710 2024-03-25T18:47:58.128 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.127+0000 7fac0a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:47:58.129 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.127+0000 7fac0a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:47:58.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.383+0000 7fac1bfcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:58.383 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.383+0000 7fac1bfcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:58.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.383+0000 7fac1bfcd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:47:58.384 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.383+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fac14137c60 con 0x7fac1413f710 2024-03-25T18:47:58.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.536+0000 7fac0a7fc640 1 -- 172.21.15.32:0/2879014794 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7fac14137c60 con 0x7fac1413f710 2024-03-25T18:47:58.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.536+0000 7fac0a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:47:58.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.536+0000 7fac0a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:47:58.539 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:47:58.539 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:47:58.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.538+0000 7fac1bfcd640 10 librados: watch_flush enter 2024-03-25T18:47:58.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.538+0000 7fac1bfcd640 10 librados: watch_flush exit 2024-03-25T18:47:58.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.538+0000 7fac1bfcd640 20 client.4606.objecter shutdown clearing up homeless session... 2024-03-25T18:47:58.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.538+0000 7fac1bfcd640 10 client.4606.objecter successfully canceled tick 2024-03-25T18:47:58.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 >> v1:172.21.15.136:6800/1141502808 conn(0x7fabf806dd10 legacy=0x7fabf80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:58.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 10 monclient: shutdown 2024-03-25T18:47:58.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:47:58.540 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 >> v1:172.21.15.136:6789/0 conn(0x7fac1413f710 legacy=0x7fac141df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:47:58.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 shutdown_connections 2024-03-25T18:47:58.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 >> 172.21.15.32:0/2879014794 conn(0x7fac140b7ea0 msgr2=0x7fac14074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:47:58.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 shutdown_connections 2024-03-25T18:47:58.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 1 -- 172.21.15.32:0/2879014794 wait complete. 2024-03-25T18:47:58.541 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:47:58.539+0000 7fac1bfcd640 1 librados: shutdown 2024-03-25T18:47:58.551 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:48:05.358 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:48:05.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.477+0000 7f5475d3b640 10 monclient: get_monmap_and_config 2024-03-25T18:48:05.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.477+0000 7f5475d3b640 10 monclient: build_initial_monmap 2024-03-25T18:48:05.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.477+0000 7f5475d3b640 10 monclient: monmap: 2024-03-25T18:48:05.477 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:05.477 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:05.477 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:48:05.477737+0000 2024-03-25T18:48:05.478 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:48:05.477737+0000 2024-03-25T18:48:05.478 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:05.478 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:05.478 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:05.478 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:48:05.478 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:48:05.478 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:05.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.478+0000 7f5475d3b640 1 Processor -- start 2024-03-25T18:48:05.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.478+0000 7f5475d3b640 1 -- start start 2024-03-25T18:48:05.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.478+0000 7f5475d3b640 10 monclient: init 2024-03-25T18:48:05.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.478+0000 7f5475d3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:05.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.478+0000 7f5475d3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:48:05.479 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.479+0000 7f5475d3b640 10 monclient(hunting): picked mon.noname-a con 0x7f54700a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:05.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.479+0000 7f5475d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f547013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:05.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.479+0000 7f5475d3b640 10 monclient(hunting): picked mon.noname-b con 0x7f54701430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:05.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.479+0000 7f5475d3b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5470142520 con 0x7f54700a98f0 2024-03-25T18:48:05.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.479+0000 7f5475d3b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54700b28a0 con 0x7f547013f710 2024-03-25T18:48:05.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.479+0000 7f5475d3b640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54700b2a30 con 0x7f54701430c0 2024-03-25T18:48:05.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.479+0000 7f5475d3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:05.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.479+0000 7f5475d3b640 10 monclient(hunting): authenticate will time out at 6591.667624s 2024-03-25T18:48:05.497 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.497+0000 7f546f7fe640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f54700a98f0 0x7f54700a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37774/0 (socket says 172.21.15.32:37774) 2024-03-25T18:48:05.497 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.497+0000 7f546f7fe640 1 -- 172.21.15.32:0/3980209974 learned_addr learned my addr 172.21.15.32:0/3980209974 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:05.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.509+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1431839799 0 0) 0x7f54700b2a30 con 0x7f54701430c0 2024-03-25T18:48:05.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.509+0000 7f546e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:05.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.509+0000 7f546e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:05.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.509+0000 7f546e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:05.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.509+0000 7f546e7fc640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:48:05.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.509+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5458003520 con 0x7f54701430c0 2024-03-25T18:48:05.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.524+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050728377 0 0) 0x7f5470142520 con 0x7f54700a98f0 2024-03-25T18:48:05.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.524+0000 7f546e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:05.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.524+0000 7f546e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:05.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.524+0000 7f546e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:05.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.524+0000 7f546e7fc640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:48:05.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.524+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54700b2a30 con 0x7f54700a98f0 2024-03-25T18:48:05.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.531+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4235024580 0 0) 0x7f54700b28a0 con 0x7f547013f710 2024-03-25T18:48:05.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.531+0000 7f546e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:05.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.531+0000 7f546e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:05.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.531+0000 7f546e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:05.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.531+0000 7f546e7fc640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:48:05.531 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.531+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5470142520 con 0x7f547013f710 2024-03-25T18:48:05.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4141822707 0 0) 0x7f54700b2a30 con 0x7f54700a98f0 2024-03-25T18:48:05.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 >> v1:172.21.15.32:6790/0 conn(0x7f54701430c0 legacy=0x7f547014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:05.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 >> v1:172.21.15.136:6789/0 conn(0x7f547013f710 legacy=0x7f54701429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:05.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:48:05.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:48:05.542 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54700b44c0 con 0x7f54700a98f0 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5460001670 con 0x7f54700a98f0 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f546e7fc640 10 monclient: dump: 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:05.543 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:05.544 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:05.544 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:05.544 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:05.544 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:05.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f5475d3b640 5 monclient: authenticate success, global_id 4632 2024-03-25T18:48:05.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.542+0000 7f5475d3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54600017d0 con 0x7f54700a98f0 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f546e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f546e7fc640 1 -- 172.21.15.32:0/3980209974 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5460001be0 con 0x7f54700a98f0 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 4 set_mon_vals no callback set 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f546e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f546e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f546e7fc640 10 monclient: dump: 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:05.576 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:05.577 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:05.577 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:05.577 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:05.577 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:05.577 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:05.577 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:05.577 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:05.578 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 10 monclient: shutdown 2024-03-25T18:48:05.578 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 1 -- 172.21.15.32:0/3980209974 >> v1:172.21.15.32:6789/0 conn(0x7f54700a98f0 legacy=0x7f54700a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 1 -- 172.21.15.32:0/3980209974 shutdown_connections 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 1 -- 172.21.15.32:0/3980209974 >> 172.21.15.32:0/3980209974 conn(0x7f54700b7ea0 msgr2=0x7f54700b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 1 -- 172.21.15.32:0/3980209974 shutdown_connections 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.576+0000 7f5475d3b640 1 -- 172.21.15.32:0/3980209974 wait complete. 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.577+0000 7f5475d3b640 10 monclient: build_initial_monmap 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.577+0000 7f5475d3b640 10 monclient: monmap: 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:48:05.579 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:48:05.580 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:05.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.578+0000 7f5475d3b640 1 librados: starting msgr at 2024-03-25T18:48:05.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.578+0000 7f5475d3b640 1 librados: starting objecter 2024-03-25T18:48:05.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.578+0000 7f5475d3b640 1 Processor -- start 2024-03-25T18:48:05.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.578+0000 7f5475d3b640 1 -- start start 2024-03-25T18:48:05.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.578+0000 7f5475d3b640 1 librados: setting wanted keys 2024-03-25T18:48:05.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.578+0000 7f5475d3b640 1 librados: calling monclient init 2024-03-25T18:48:05.580 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.578+0000 7f5475d3b640 10 monclient: init 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 10 monclient(hunting): picked mon.noname-a con 0x7f54700a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 10 monclient(hunting): picked mon.noname-b con 0x7f547013f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 10 monclient(hunting): picked mon.noname-c con 0x7f54701430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54701dad60 con 0x7f54700a98f0 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54700b1730 con 0x7f547013f710 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f547014de50 con 0x7f54701430c0 2024-03-25T18:48:05.581 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:05.582 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.579+0000 7f5475d3b640 10 monclient(hunting): authenticate will time out at 6591.768275s 2024-03-25T18:48:05.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.588+0000 7f546ffff640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f54701430c0 0x7f54701e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:53106/0 (socket says 172.21.15.32:53106) 2024-03-25T18:48:05.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.588+0000 7f546ffff640 1 -- 172.21.15.32:0/1114366995 learned_addr learned my addr 172.21.15.32:0/1114366995 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:05.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.593+0000 7f5457fff640 10 client.?.objecter ms_handle_connect 0x7f547013f710 2024-03-25T18:48:05.592 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.593+0000 7f5457fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:05.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.593+0000 7f5457fff640 10 client.?.objecter ms_handle_connect 0x7f54701430c0 2024-03-25T18:48:05.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.593+0000 7f5457fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:05.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.595+0000 7f5457fff640 10 client.?.objecter ms_handle_connect 0x7f54700a98f0 2024-03-25T18:48:05.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.595+0000 7f5457fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:05.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.597+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 489146917 0 0) 0x7f54700b1730 con 0x7f547013f710 2024-03-25T18:48:05.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.597+0000 7f5457fff640 10 monclient(hunting): _negotiate 2024-03-25T18:48:05.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.597+0000 7f5457fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:05.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.597+0000 7f5457fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:05.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.598+0000 7f5457fff640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:48:05.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.598+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f545c004eb0 con 0x7f547013f710 2024-03-25T18:48:05.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.607+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1212764218 0 0) 0x7f54701dad60 con 0x7f54700a98f0 2024-03-25T18:48:05.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.607+0000 7f5457fff640 10 monclient(hunting): _negotiate 2024-03-25T18:48:05.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.607+0000 7f5457fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:05.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.607+0000 7f5457fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:05.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.607+0000 7f5457fff640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:48:05.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.607+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54700b1730 con 0x7f54700a98f0 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.608+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2797220642 0 0) 0x7f545c004eb0 con 0x7f547013f710 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.608+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54701dad60 con 0x7f547013f710 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.608+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f54640018f0 con 0x7f547013f710 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.608+0000 7f5457fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.608+0000 7f5457fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.608+0000 7f5457fff640 10 monclient(hunting): dump: 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:05.609 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:05.610 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:05.610 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:05.610 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:05.610 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.608+0000 7f5457fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3025500397 0 0) 0x7f54701dad60 con 0x7f547013f710 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 >> v1:172.21.15.32:6790/0 conn(0x7f54701430c0 legacy=0x7f54701e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 >> v1:172.21.15.32:6789/0 conn(0x7f54700a98f0 legacy=0x7f54701db1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 10 monclient: _finish_hunting 0 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 1 monclient: found mon.b 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547014eb60 con 0x7f547013f710 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 10 monclient: _finish_auth 0 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 10 monclient: _check_auth_tickets 2024-03-25T18:48:05.620 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5457fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 10 monclient: _renew_subs 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54701da8f0 con 0x7f547013f710 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 10 monclient: _renew_subs 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f547014e2a0 con 0x7f547013f710 2024-03-25T18:48:05.621 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.619+0000 7f5475d3b640 1 librados: init done 2024-03-25T18:48:05.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.620+0000 7f5475d3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:05.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.620+0000 7f5475d3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:05.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.620+0000 7f5475d3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:05.622 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.620+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54700b0a40 con 0x7f547013f710 2024-03-25T18:48:05.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.639+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5464003610 con 0x7f547013f710 2024-03-25T18:48:05.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.639+0000 7f5457fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:05.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.639+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f54640039a0 con 0x7f547013f710 2024-03-25T18:48:05.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.639+0000 7f5457fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.639+0000 7f5457fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.639+0000 7f5457fff640 10 monclient: dump: 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:05.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.639+0000 7f546dffb640 4 set_mon_vals no callback set 2024-03-25T18:48:05.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.640+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5464003bc0 con 0x7f547013f710 2024-03-25T18:48:05.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.640+0000 7f546f7fe640 10 monclient: get_auth_request con 0x7f545c06dd10 auth_method 0 2024-03-25T18:48:05.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.640+0000 7f5457fff640 10 client.4612.objecter ms_handle_connect 0x7f545c06dd10 2024-03-25T18:48:05.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.643+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5464053600 con 0x7f547013f710 2024-03-25T18:48:05.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.643+0000 7f5457fff640 10 client.4612.objecter ms_dispatch 0x7f5470000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:48:05.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.643+0000 7f5457fff640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:48:05.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.643+0000 7f5457fff640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:48:05.643 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.643+0000 7f5457fff640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T18:48:05.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.897+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f546404f110 con 0x7f547013f710 2024-03-25T18:48:05.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.897+0000 7f5457fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:05.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:05.897+0000 7f5457fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:48:06.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.153+0000 7f5475d3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:06.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.153+0000 7f5475d3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:06.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.153+0000 7f5475d3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:06.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.153+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54700627b0 con 0x7f547013f710 2024-03-25T18:48:06.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.179+0000 7f5457fff640 1 -- 172.21.15.32:0/1114366995 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5464051040 con 0x7f547013f710 2024-03-25T18:48:06.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.179+0000 7f5457fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:06.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.179+0000 7f5457fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:48:06.179 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:48:06.179 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 10 librados: watch_flush enter 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 10 librados: watch_flush exit 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 10 client.4612.objecter successfully canceled tick 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 >> v1:172.21.15.136:6800/1141502808 conn(0x7f545c06dd10 legacy=0x7f545c0701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 10 monclient: shutdown 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 >> v1:172.21.15.136:6789/0 conn(0x7f547013f710 legacy=0x7f54700b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:06.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 shutdown_connections 2024-03-25T18:48:06.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 >> 172.21.15.32:0/1114366995 conn(0x7f54700b7ea0 msgr2=0x7f5470074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:06.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 shutdown_connections 2024-03-25T18:48:06.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 1 -- 172.21.15.32:0/1114366995 wait complete. 2024-03-25T18:48:06.183 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:06.181+0000 7f5475d3b640 1 librados: shutdown 2024-03-25T18:48:06.192 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:48:12.999 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.114+0000 7f016cea8640 10 monclient: get_monmap_and_config 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.114+0000 7f016cea8640 10 monclient: build_initial_monmap 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.114+0000 7f016cea8640 10 monclient: monmap: 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:48:13.115327+0000 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:48:13.115327+0000 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:13.115 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:48:13.116 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:48:13.116 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:13.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.115+0000 7f016cea8640 1 Processor -- start 2024-03-25T18:48:13.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.115+0000 7f016cea8640 1 -- start start 2024-03-25T18:48:13.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.115+0000 7f016cea8640 10 monclient: init 2024-03-25T18:48:13.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:13.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:48:13.117 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 10 monclient(hunting): picked mon.noname-c con 0x7f01680a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:13.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 10 monclient(hunting): picked mon.noname-b con 0x7f016813f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:13.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 10 monclient(hunting): picked mon.noname-a con 0x7f01681430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:13.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168142520 con 0x7f01681430c0 2024-03-25T18:48:13.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01680b28a0 con 0x7f01680a98f0 2024-03-25T18:48:13.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01680b2a30 con 0x7f016813f710 2024-03-25T18:48:13.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:13.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.116+0000 7f016cea8640 10 monclient(hunting): authenticate will time out at 6599.305352s 2024-03-25T18:48:13.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.125+0000 7f0166d9d640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f01681430c0 0x7f016814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:46630/0 (socket says 172.21.15.32:46630) 2024-03-25T18:48:13.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.125+0000 7f0166d9d640 1 -- 172.21.15.32:0/3441560538 learned_addr learned my addr 172.21.15.32:0/3441560538 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:13.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.144+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4263929104 0 0) 0x7f01680b2a30 con 0x7f016813f710 2024-03-25T18:48:13.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.144+0000 7f0165d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:48:13.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.144+0000 7f0165d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:13.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.144+0000 7f0165d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:13.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.144+0000 7f0165d9b640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:48:13.144 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.144+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0154003520 con 0x7f016813f710 2024-03-25T18:48:13.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.145+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770231027 0 0) 0x7f0168142520 con 0x7f01681430c0 2024-03-25T18:48:13.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.145+0000 7f0165d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:48:13.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.145+0000 7f0165d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:13.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.145+0000 7f0165d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:13.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.145+0000 7f0165d9b640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:48:13.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.145+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01680b2a30 con 0x7f01681430c0 2024-03-25T18:48:13.155 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.155+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3030650166 0 0) 0x7f01680b28a0 con 0x7f01680a98f0 2024-03-25T18:48:13.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.155+0000 7f0165d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:48:13.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.155+0000 7f0165d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:13.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.155+0000 7f0165d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:13.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.155+0000 7f0165d9b640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:48:13.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.155+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0168142520 con 0x7f01680a98f0 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4562240 0 0) 0x7f0154003520 con 0x7f016813f710 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 >> v1:172.21.15.136:6789/0 conn(0x7f01680a98f0 legacy=0x7f01680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 >> v1:172.21.15.32:6789/0 conn(0x7f01681430c0 legacy=0x7f016814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 1 monclient: found mon.noname-b 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01680b44c0 con 0x7f016813f710 2024-03-25T18:48:13.163 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 10 monclient: _finish_auth 0 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0150001690 con 0x7f016813f710 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f016cea8640 5 monclient: authenticate success, global_id 4613 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f016cea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f0165d9b640 10 monclient: dump: 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:13.164 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:13.165 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:13.165 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:13.165 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:13.165 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:13.165 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.162+0000 7f016cea8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:13.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.195+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f01500017f0 con 0x7f016813f710 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.195+0000 7f0165d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.195+0000 7f0165d9b640 1 -- 172.21.15.32:0/3441560538 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0150001c00 con 0x7f016813f710 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.195+0000 7f0165d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.195+0000 7f0165d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.195+0000 7f0165d9b640 10 monclient: dump: 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:13.196 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:13.197 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:13.197 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:13.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.195+0000 7f016cea8640 10 monclient: get_monmap_and_config success 2024-03-25T18:48:13.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.195+0000 7f016cea8640 4 set_mon_vals no callback set 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.196+0000 7f016cea8640 10 monclient: shutdown 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.196+0000 7f016cea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.196+0000 7f016cea8640 1 -- 172.21.15.32:0/3441560538 >> v1:172.21.15.32:6790/0 conn(0x7f016813f710 legacy=0x7f01681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.196+0000 7f016cea8640 1 -- 172.21.15.32:0/3441560538 shutdown_connections 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.196+0000 7f016cea8640 1 -- 172.21.15.32:0/3441560538 >> 172.21.15.32:0/3441560538 conn(0x7f01680b7ea0 msgr2=0x7f01680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.196+0000 7f016cea8640 1 -- 172.21.15.32:0/3441560538 shutdown_connections 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.196+0000 7f016cea8640 1 -- 172.21.15.32:0/3441560538 wait complete. 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.197+0000 7f016cea8640 10 monclient: build_initial_monmap 2024-03-25T18:48:13.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.197+0000 7f016cea8640 10 monclient: monmap: 2024-03-25T18:48:13.200 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:13.200 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:48:13.200 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:48:13.200 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.197+0000 7f016cea8640 1 librados: starting msgr at 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.197+0000 7f016cea8640 1 librados: starting objecter 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.197+0000 7f016cea8640 1 Processor -- start 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 1 -- start start 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 1 librados: setting wanted keys 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 1 librados: calling monclient init 2024-03-25T18:48:13.201 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 10 monclient: init 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 10 monclient(hunting): picked mon.noname-a con 0x7f01680a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 10 monclient(hunting): picked mon.noname-b con 0x7f016813f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 10 monclient(hunting): picked mon.noname-c con 0x7f01681430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01681370a0 con 0x7f01680a98f0 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168135090 con 0x7f016813f710 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0168135220 con 0x7f01681430c0 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:13.202 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.198+0000 7f016cea8640 10 monclient(hunting): authenticate will time out at 6599.387432s 2024-03-25T18:48:13.204 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.204+0000 7f0166d9d640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f01681430c0 0x7f01681d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:43126/0 (socket says 172.21.15.32:43126) 2024-03-25T18:48:13.205 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.204+0000 7f0166d9d640 1 -- 172.21.15.32:0/43798807 learned_addr learned my addr 172.21.15.32:0/43798807 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:13.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.208+0000 7f015effd640 10 client.?.objecter ms_handle_connect 0x7f01681430c0 2024-03-25T18:48:13.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.208+0000 7f015effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:13.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.210+0000 7f015effd640 10 client.?.objecter ms_handle_connect 0x7f01680a98f0 2024-03-25T18:48:13.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.210+0000 7f015effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:13.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.211+0000 7f015effd640 10 client.?.objecter ms_handle_connect 0x7f016813f710 2024-03-25T18:48:13.211 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.211+0000 7f015effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:13.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.217+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791153383 0 0) 0x7f01681370a0 con 0x7f01680a98f0 2024-03-25T18:48:13.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.217+0000 7f015effd640 10 monclient(hunting): _negotiate 2024-03-25T18:48:13.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.217+0000 7f015effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:13.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.217+0000 7f015effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:13.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.217+0000 7f015effd640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:48:13.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.217+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0148004eb0 con 0x7f01680a98f0 2024-03-25T18:48:13.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961636469 0 0) 0x7f0168135090 con 0x7f016813f710 2024-03-25T18:48:13.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 10 monclient(hunting): _negotiate 2024-03-25T18:48:13.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:13.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:13.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:48:13.220 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01681370a0 con 0x7f016813f710 2024-03-25T18:48:13.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2592364240 0 0) 0x7f0168135220 con 0x7f01681430c0 2024-03-25T18:48:13.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 10 monclient(hunting): _negotiate 2024-03-25T18:48:13.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:13.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:13.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:48:13.221 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.220+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0168135090 con 0x7f01681430c0 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.229+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2976625412 0 0) 0x7f01681370a0 con 0x7f016813f710 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.229+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0168135220 con 0x7f016813f710 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.229+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0150001b00 con 0x7f016813f710 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.229+0000 7f015effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.229+0000 7f015effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.229+0000 7f015effd640 10 monclient(hunting): dump: 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.230 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:13.231 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:13.231 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:13.231 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:13.231 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:13.231 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:13.231 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.229+0000 7f015effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:48:13.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.233+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3301677413 0 0) 0x7f0148004eb0 con 0x7f01680a98f0 2024-03-25T18:48:13.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.233+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01681370a0 con 0x7f01680a98f0 2024-03-25T18:48:13.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.233+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f01580018f0 con 0x7f01680a98f0 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.233+0000 7f015effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.233+0000 7f015effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.233+0000 7f015effd640 10 monclient(hunting): dump: 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:13.234 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 866097459 0 0) 0x7f0168135220 con 0x7f016813f710 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 >> v1:172.21.15.32:6790/0 conn(0x7f01681430c0 legacy=0x7f01681d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 >> v1:172.21.15.32:6789/0 conn(0x7f01680a98f0 legacy=0x7f01681dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 10 monclient: _finish_hunting 0 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 1 monclient: found mon.b 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01681d5000 con 0x7f016813f710 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 10 monclient: _finish_auth 0 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 10 monclient: _check_auth_tickets 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f015effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:13.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f016cea8640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:48:13.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f016cea8640 10 monclient: _renew_subs 2024-03-25T18:48:13.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f016cea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:13.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01681d52b0 con 0x7f016813f710 2024-03-25T18:48:13.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f016cea8640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:48:13.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.244+0000 7f016cea8640 10 monclient: _renew_subs 2024-03-25T18:48:13.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.245+0000 7f016cea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:13.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.245+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0168135910 con 0x7f016813f710 2024-03-25T18:48:13.246 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.245+0000 7f016cea8640 1 librados: init done 2024-03-25T18:48:13.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.246+0000 7f016cea8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:13.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.246+0000 7f016cea8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:13.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.246+0000 7f016cea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:13.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.246+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01680627b0 con 0x7f016813f710 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.519+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0150001ca0 con 0x7f016813f710 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.519+0000 7f015effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.519+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0150003a40 con 0x7f016813f710 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.519+0000 7f015effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.519+0000 7f015effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.519+0000 7f015effd640 10 monclient: dump: 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:13.520 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:13.521 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:13.521 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:13.521 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:13.521 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:13.521 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:13.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.519+0000 7f016559a640 4 set_mon_vals no callback set 2024-03-25T18:48:13.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.520+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f0150019a10 con 0x7f016813f710 2024-03-25T18:48:13.521 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.521+0000 7f016659c640 10 monclient: get_auth_request con 0x7f014806df70 auth_method 0 2024-03-25T18:48:13.522 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.521+0000 7f015effd640 10 client.4618.objecter ms_handle_connect 0x7f014806df70 2024-03-25T18:48:13.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.525+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f01500535d0 con 0x7f016813f710 2024-03-25T18:48:13.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.525+0000 7f015effd640 10 client.4618.objecter ms_dispatch 0x7f0168000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:48:13.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.525+0000 7f015effd640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:48:13.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.525+0000 7f015effd640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:48:13.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.526+0000 7f015effd640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T18:48:13.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.526+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f015004f0e0 con 0x7f016813f710 2024-03-25T18:48:13.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.526+0000 7f015effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:13.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.526+0000 7f015effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:48:13.771 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.770+0000 7f016cea8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:13.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.770+0000 7f016cea8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:13.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.770+0000 7f016cea8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:13.772 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.770+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f016814f240 con 0x7f016813f710 2024-03-25T18:48:13.793 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:26.121 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:13.803 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:26.131 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:13.839 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:26.167 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:13.859 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:26.187 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:13.861 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:26.178 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:13.864 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:26.192 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:13.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.865+0000 7f015effd640 1 -- 172.21.15.32:0/43798807 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f0150051010 con 0x7f016813f710 2024-03-25T18:48:13.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.865+0000 7f015effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:13.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.865+0000 7f015effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:48:13.867 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:48:13.867 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:48:13.867 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:26.183 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 10 librados: watch_flush enter 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 10 librados: watch_flush exit 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 10 client.4618.objecter successfully canceled tick 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 >> v1:172.21.15.136:6800/1141502808 conn(0x7f014806df70 legacy=0x7f0148070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 10 monclient: shutdown 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 >> v1:172.21.15.136:6789/0 conn(0x7f016813f710 legacy=0x7f01681d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 shutdown_connections 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 >> 172.21.15.32:0/43798807 conn(0x7f01680b7ea0 msgr2=0x7f0168074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 shutdown_connections 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 1 -- 172.21.15.32:0/43798807 wait complete. 2024-03-25T18:48:13.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:13.867+0000 7f016cea8640 1 librados: shutdown 2024-03-25T18:48:13.879 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:48:13.889 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:26.206 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:13.895 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:26.223 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:20.685 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:48:20.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.805+0000 7f16b404a640 10 monclient: get_monmap_and_config 2024-03-25T18:48:20.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.805+0000 7f16b404a640 10 monclient: build_initial_monmap 2024-03-25T18:48:20.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.805+0000 7f16b404a640 10 monclient: monmap: 2024-03-25T18:48:20.806 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:20.806 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:20.806 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:48:20.806732+0000 2024-03-25T18:48:20.806 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:48:20.806732+0000 2024-03-25T18:48:20.807 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:20.807 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:20.807 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:20.807 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:48:20.807 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:48:20.807 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:20.808 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.805+0000 7f16b404a640 1 Processor -- start 2024-03-25T18:48:20.808 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.806+0000 7f16b404a640 1 -- start start 2024-03-25T18:48:20.808 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.806+0000 7f16b404a640 10 monclient: init 2024-03-25T18:48:20.808 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.806+0000 7f16b404a640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:20.808 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.806+0000 7f16b404a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:48:20.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.806+0000 7f16b404a640 10 monclient(hunting): picked mon.noname-c con 0x7f16ac0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:20.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.806+0000 7f16b404a640 10 monclient(hunting): picked mon.noname-b con 0x7f16ac13f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:20.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.807+0000 7f16b404a640 10 monclient(hunting): picked mon.noname-a con 0x7f16ac1430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:20.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.807+0000 7f16b404a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac142520 con 0x7f16ac1430c0 2024-03-25T18:48:20.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.807+0000 7f16b404a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0b28a0 con 0x7f16ac0a98f0 2024-03-25T18:48:20.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.807+0000 7f16b404a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac0b2a30 con 0x7f16ac13f710 2024-03-25T18:48:20.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.807+0000 7f16b404a640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:20.809 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.807+0000 7f16b404a640 10 monclient(hunting): authenticate will time out at 6606.996804s 2024-03-25T18:48:20.815 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.814+0000 7f16b15e5640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f16ac13f710 0x7f16ac1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:47240/0 (socket says 172.21.15.32:47240) 2024-03-25T18:48:20.815 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.814+0000 7f16b15e5640 1 -- 172.21.15.32:0/3104173418 learned_addr learned my addr 172.21.15.32:0/3104173418 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:20.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.821+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691806300 0 0) 0x7f16ac0b28a0 con 0x7f16ac0a98f0 2024-03-25T18:48:20.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.821+0000 7f16b0de4640 10 monclient(hunting): _negotiate 2024-03-25T18:48:20.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.821+0000 7f16b0de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:20.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.821+0000 7f16b0de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:20.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.821+0000 7f16b0de4640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:48:20.823 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.822+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1698003520 con 0x7f16ac0a98f0 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1839364435 0 0) 0x7f16ac0b2a30 con 0x7f16ac13f710 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient(hunting): _negotiate 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ac0b28a0 con 0x7f16ac13f710 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2226074245 0 0) 0x7f1698003520 con 0x7f16ac0a98f0 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 >> v1:172.21.15.32:6790/0 conn(0x7f16ac13f710 legacy=0x7f16ac1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 >> v1:172.21.15.32:6789/0 conn(0x7f16ac1430c0 legacy=0x7f16ac14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient: _finish_hunting 0 2024-03-25T18:48:20.831 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 1 monclient: found mon.noname-c 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac0b44c0 con 0x7f16ac0a98f0 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient: _finish_auth 0 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient: _check_auth_tickets 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f16a8001690 con 0x7f16ac0a98f0 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b0de4640 10 monclient: dump: 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:20.832 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:20.833 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:20.833 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:20.833 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:20.833 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:20.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b404a640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:48:20.833 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.829+0000 7f16b404a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:20.837 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.836+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f16a80017f0 con 0x7f16ac0a98f0 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.836+0000 7f16b0de4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.836+0000 7f16b0de4640 1 -- 172.21.15.32:0/3104173418 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f16a8001c00 con 0x7f16ac0a98f0 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.836+0000 7f16b0de4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.836+0000 7f16b0de4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.836+0000 7f16b0de4640 10 monclient: dump: 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:20.838 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:20.839 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:20.839 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:20.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.837+0000 7f16b404a640 10 monclient: get_monmap_and_config success 2024-03-25T18:48:20.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.837+0000 7f16b404a640 4 set_mon_vals no callback set 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.837+0000 7f16b404a640 10 monclient: shutdown 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.837+0000 7f16b404a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.837+0000 7f16b404a640 1 -- 172.21.15.32:0/3104173418 >> v1:172.21.15.136:6789/0 conn(0x7f16ac0a98f0 legacy=0x7f16ac0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.838+0000 7f16b404a640 1 -- 172.21.15.32:0/3104173418 shutdown_connections 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.838+0000 7f16b404a640 1 -- 172.21.15.32:0/3104173418 >> 172.21.15.32:0/3104173418 conn(0x7f16ac0b7ea0 msgr2=0x7f16ac0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.838+0000 7f16b404a640 1 -- 172.21.15.32:0/3104173418 shutdown_connections 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.838+0000 7f16b404a640 1 -- 172.21.15.32:0/3104173418 wait complete. 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.838+0000 7f16b404a640 10 monclient: build_initial_monmap 2024-03-25T18:48:20.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.838+0000 7f16b404a640 10 monclient: monmap: 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 librados: starting msgr at 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 librados: starting objecter 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 Processor -- start 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 -- start start 2024-03-25T18:48:20.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 librados: setting wanted keys 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 librados: calling monclient init 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 10 monclient: init 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 10 monclient(hunting): picked mon.noname-a con 0x7f16ac0a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 10 monclient(hunting): picked mon.noname-b con 0x7f16ac13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 10 monclient(hunting): picked mon.noname-c con 0x7f16ac1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac135730 con 0x7f16ac0a98f0 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac1358c0 con 0x7f16ac13f710 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16ac135a50 con 0x7f16ac1430c0 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:20.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.839+0000 7f16b404a640 10 monclient(hunting): authenticate will time out at 6607.029415s 2024-03-25T18:48:20.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.860+0000 7f16b25e7640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f16ac1430c0 0x7f16ac1d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:47266/0 (socket says 172.21.15.32:47266) 2024-03-25T18:48:20.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.860+0000 7f16b25e7640 1 -- 172.21.15.32:0/265247781 learned_addr learned my addr 172.21.15.32:0/265247781 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:20.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.861+0000 7f16a27fc640 10 client.?.objecter ms_handle_connect 0x7f16ac1430c0 2024-03-25T18:48:20.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.861+0000 7f16a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:20.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.864+0000 7f16a27fc640 10 client.?.objecter ms_handle_connect 0x7f16ac13f710 2024-03-25T18:48:20.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.864+0000 7f16a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:20.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.867+0000 7f16a27fc640 10 client.?.objecter ms_handle_connect 0x7f16ac0a98f0 2024-03-25T18:48:20.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.867+0000 7f16a27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:20.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.875+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1745981588 0 0) 0x7f16ac135a50 con 0x7f16ac1430c0 2024-03-25T18:48:20.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.875+0000 7f16a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:20.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.875+0000 7f16a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:20.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.875+0000 7f16a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:20.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.875+0000 7f16a27fc640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:48:20.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.875+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1690004eb0 con 0x7f16ac1430c0 2024-03-25T18:48:20.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.876+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058779816 0 0) 0x7f16ac1358c0 con 0x7f16ac13f710 2024-03-25T18:48:20.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.876+0000 7f16a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:20.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.876+0000 7f16a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:20.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.876+0000 7f16a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:20.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.876+0000 7f16a27fc640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:48:20.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.876+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ac135a50 con 0x7f16ac13f710 2024-03-25T18:48:20.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.885+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3988016671 0 0) 0x7f16ac135730 con 0x7f16ac0a98f0 2024-03-25T18:48:20.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.885+0000 7f16a27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:20.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.885+0000 7f16a27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:20.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.885+0000 7f16a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:20.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.885+0000 7f16a27fc640 10 monclient(hunting): my global_id is 4647 2024-03-25T18:48:20.886 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.885+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f16ac1358c0 con 0x7f16ac0a98f0 2024-03-25T18:48:20.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.903+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1626577660 0 0) 0x7f16ac1358c0 con 0x7f16ac0a98f0 2024-03-25T18:48:20.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.903+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16ac135730 con 0x7f16ac0a98f0 2024-03-25T18:48:20.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.903+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f16a8001b00 con 0x7f16ac0a98f0 2024-03-25T18:48:20.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.903+0000 7f16a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:20.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.903+0000 7f16a27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:48:20.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.903+0000 7f16a27fc640 10 monclient(hunting): dump: 2024-03-25T18:48:20.904 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:20.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.903+0000 7f16a27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.905+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2780895881 0 0) 0x7f1690004eb0 con 0x7f16ac1430c0 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.905+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f16ac1358c0 con 0x7f16ac1430c0 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.905+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f16a4001c90 con 0x7f16ac1430c0 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.905+0000 7f16a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.905+0000 7f16a27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.905+0000 7f16a27fc640 10 monclient(hunting): dump: 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.906 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.907 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:20.907 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:20.907 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:20.907 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:20.907 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:20.907 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:20.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.911+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 96263805 0 0) 0x7f16ac135a50 con 0x7f16ac13f710 2024-03-25T18:48:20.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.911+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1690004eb0 con 0x7f16ac13f710 2024-03-25T18:48:20.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.911+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f169c001930 con 0x7f16ac13f710 2024-03-25T18:48:20.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.911+0000 7f16a27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:20.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.911+0000 7f16a27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:20.912 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.911+0000 7f16a27fc640 10 monclient(hunting): dump: 2024-03-25T18:48:20.912 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:20.912 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:20.913 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.913 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:20.913 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:20.913 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:20.913 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:20.913 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:20.913 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:20.913 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:20.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.925+0000 7f16b25e7640 1 -- 172.21.15.32:0/265247781 >> v1:172.21.15.32:6790/0 conn(0x7f16ac1430c0 legacy=0x7f16ac1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:48:20.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.925+0000 7f16b25e7640 1 -- 172.21.15.32:0/265247781 >> v1:172.21.15.32:6790/0 conn(0x7f16ac1430c0 legacy=0x7f16ac1d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:48:20.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.925+0000 7f16b25e7640 1 --1- 172.21.15.32:0/265247781 >> v1:172.21.15.32:6790/0 conn(0x7f16ac1430c0 0x7f16ac1d48f0 :-1 s=OPENED pgs=66346 cs=1 l=1).handle_message read tag failed 2024-03-25T18:48:20.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.925+0000 7f16b25e7640 1 --1- 172.21.15.32:0/265247781 >> v1:172.21.15.32:6790/0 conn(0x7f16ac1430c0 0x7f16ac1d48f0 :-1 s=OPENED pgs=66346 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:48:20.927 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.925+0000 7f16a27fc640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.32:6790/0 2024-03-25T18:48:20.931 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2598179163 0 0) 0x7f1690004eb0 con 0x7f16ac13f710 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 >> v1:172.21.15.32:6790/0 conn(0x7f16ac1430c0 legacy=0x7f16ac1d48f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 >> v1:172.21.15.32:6789/0 conn(0x7f16ac0a98f0 legacy=0x7f16ac1dae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 1 monclient: found mon.b 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16ac1d5000 con 0x7f16ac13f710 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 10 monclient: _finish_auth 0 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 10 monclient: _renew_subs 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16ac1d52b0 con 0x7f16ac13f710 2024-03-25T18:48:20.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 10 client.4624.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:48:20.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 10 monclient: _renew_subs 2024-03-25T18:48:20.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:20.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f16ac13c900 con 0x7f16ac13f710 2024-03-25T18:48:20.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.930+0000 7f16b404a640 1 librados: init done 2024-03-25T18:48:20.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.931+0000 7f16b404a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:20.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.931+0000 7f16b404a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:20.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.931+0000 7f16b404a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:20.933 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:20.931+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16ac13cbb0 con 0x7f16ac13f710 2024-03-25T18:48:21.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.206+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f169c001d40 con 0x7f16ac13f710 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.206+0000 7f16a27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.206+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f169c0047e0 con 0x7f16ac13f710 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.206+0000 7f16a27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.206+0000 7f16a27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.206+0000 7f16a27fc640 10 monclient: dump: 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:21.208 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:21.209 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:21.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.206+0000 7f16a3fff640 4 set_mon_vals no callback set 2024-03-25T18:48:21.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.207+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f169c004a00 con 0x7f16ac13f710 2024-03-25T18:48:21.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.208+0000 7f16b1de6640 10 monclient: get_auth_request con 0x7f169006e160 auth_method 0 2024-03-25T18:48:21.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.208+0000 7f16a27fc640 10 client.4624.objecter ms_handle_connect 0x7f169006e160 2024-03-25T18:48:21.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.213+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f169c054590 con 0x7f16ac13f710 2024-03-25T18:48:21.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.213+0000 7f16a27fc640 10 client.4624.objecter ms_dispatch 0x7f16ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:48:21.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.213+0000 7f16a27fc640 3 client.4624.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:48:21.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.213+0000 7f16a27fc640 3 client.4624.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:48:21.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.213+0000 7f16a27fc640 20 client.4624.objecter dump_active .. 0 homeless 2024-03-25T18:48:21.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.213+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f169c04fed0 con 0x7f16ac13f710 2024-03-25T18:48:21.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.213+0000 7f16a27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:21.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.213+0000 7f16a27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:48:21.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.448+0000 7f16b404a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:21.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.448+0000 7f16b404a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:21.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.448+0000 7f16b404a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:21.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.448+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f16ac1371c0 con 0x7f16ac13f710 2024-03-25T18:48:21.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.471+0000 7f16a27fc640 1 -- 172.21.15.32:0/265247781 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f169c051fd0 con 0x7f16ac13f710 2024-03-25T18:48:21.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.471+0000 7f16a27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:21.472 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.471+0000 7f16a27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:48:21.472 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:48:21.473 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:48:21.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.473+0000 7f16b404a640 10 librados: watch_flush enter 2024-03-25T18:48:21.475 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.473+0000 7f16b404a640 10 librados: watch_flush exit 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.473+0000 7f16b404a640 20 client.4624.objecter shutdown clearing up homeless session... 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.473+0000 7f16b404a640 10 client.4624.objecter successfully canceled tick 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 >> v1:172.21.15.136:6800/1141502808 conn(0x7f169006e160 legacy=0x7f1690070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 10 monclient: shutdown 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 >> v1:172.21.15.136:6789/0 conn(0x7f16ac13f710 legacy=0x7f16ac1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 shutdown_connections 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 >> 172.21.15.32:0/265247781 conn(0x7f16ac0b7ea0 msgr2=0x7f16ac14ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 shutdown_connections 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 1 -- 172.21.15.32:0/265247781 wait complete. 2024-03-25T18:48:21.476 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:21.474+0000 7f16b404a640 1 librados: shutdown 2024-03-25T18:48:21.487 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:48:28.295 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:48:28.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.395+0000 7f5225584640 10 monclient: get_monmap_and_config 2024-03-25T18:48:28.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.395+0000 7f5225584640 10 monclient: build_initial_monmap 2024-03-25T18:48:28.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.395+0000 7f5225584640 10 monclient: monmap: 2024-03-25T18:48:28.396 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:28.396 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:28.397 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:48:28.396825+0000 2024-03-25T18:48:28.397 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:48:28.396825+0000 2024-03-25T18:48:28.397 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:28.397 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:28.397 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:28.397 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:48:28.397 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:48:28.397 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:28.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 1 Processor -- start 2024-03-25T18:48:28.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 1 -- start start 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 10 monclient: init 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 10 monclient(hunting): picked mon.noname-b con 0x7f52200a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 10 monclient(hunting): picked mon.noname-a con 0x7f522013f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 10 monclient(hunting): picked mon.noname-c con 0x7f52201430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5220142520 con 0x7f522013f710 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52200b28a0 con 0x7f52201430c0 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52200b2a30 con 0x7f52200a98f0 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:28.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.396+0000 7f5225584640 10 monclient(hunting): authenticate will time out at 6614.586492s 2024-03-25T18:48:28.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.411+0000 7f5217fff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f522013f710 0x7f52201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44778/0 (socket says 172.21.15.32:44778) 2024-03-25T18:48:28.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.411+0000 7f5217fff640 1 -- 172.21.15.32:0/838974621 learned_addr learned my addr 172.21.15.32:0/838974621 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.424+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106447365 0 0) 0x7f5220142520 con 0x7f522013f710 2024-03-25T18:48:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.424+0000 7f521e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.424+0000 7f521e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.424+0000 7f521e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.424+0000 7f521e7fc640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:48:28.425 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.424+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f520c003520 con 0x7f522013f710 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3022291489 0 0) 0x7f520c003520 con 0x7f522013f710 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 >> v1:172.21.15.32:6790/0 conn(0x7f52200a98f0 legacy=0x7f52200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 >> v1:172.21.15.136:6789/0 conn(0x7f52201430c0 legacy=0x7f522014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52200b44c0 con 0x7f522013f710 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5208001690 con 0x7f522013f710 2024-03-25T18:48:28.442 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f521e7fc640 10 monclient: dump: 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f5225584640 5 monclient: authenticate success, global_id 4650 2024-03-25T18:48:28.443 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.440+0000 7f5225584640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:28.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.451+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52080017f0 con 0x7f522013f710 2024-03-25T18:48:28.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.451+0000 7f521e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:28.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.451+0000 7f521e7fc640 1 -- 172.21.15.32:0/838974621 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5208001c00 con 0x7f522013f710 2024-03-25T18:48:28.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.451+0000 7f5225584640 10 monclient: get_monmap_and_config success 2024-03-25T18:48:28.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.451+0000 7f5225584640 4 set_mon_vals no callback set 2024-03-25T18:48:28.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.451+0000 7f521e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:28.452 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.451+0000 7f521e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.451+0000 7f521e7fc640 10 monclient: dump: 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:28.453 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.452+0000 7f5225584640 10 monclient: shutdown 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.452+0000 7f5225584640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.452+0000 7f5225584640 1 -- 172.21.15.32:0/838974621 >> v1:172.21.15.32:6789/0 conn(0x7f522013f710 legacy=0x7f52201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.452+0000 7f5225584640 1 -- 172.21.15.32:0/838974621 shutdown_connections 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.452+0000 7f5225584640 1 -- 172.21.15.32:0/838974621 >> 172.21.15.32:0/838974621 conn(0x7f52200b7ea0 msgr2=0x7f52200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.452+0000 7f5225584640 1 -- 172.21.15.32:0/838974621 shutdown_connections 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.452+0000 7f5225584640 1 -- 172.21.15.32:0/838974621 wait complete. 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 10 monclient: build_initial_monmap 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 10 monclient: monmap: 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:48:28.455 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 1 librados: starting msgr at 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 1 librados: starting objecter 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 1 Processor -- start 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 1 -- start start 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 1 librados: setting wanted keys 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 1 librados: calling monclient init 2024-03-25T18:48:28.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.453+0000 7f5225584640 10 monclient: init 2024-03-25T18:48:28.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:28.457 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 10 monclient(hunting): picked mon.noname-a con 0x7f52200a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 10 monclient(hunting): picked mon.noname-c con 0x7f522013f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 10 monclient(hunting): picked mon.noname-b con 0x7f52201430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5220138980 con 0x7f52200a98f0 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5220138640 con 0x7f52201430c0 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52200b1700 con 0x7f522013f710 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.454+0000 7f5225584640 10 monclient(hunting): authenticate will time out at 6614.644252s 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.455+0000 7f521f7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f52201430c0 0x7f52201db000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:49500/0 (socket says 172.21.15.32:49500) 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.455+0000 7f521f7fe640 1 -- 172.21.15.32:0/2701765659 learned_addr learned my addr 172.21.15.32:0/2701765659 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.456+0000 7f52177fe640 10 client.?.objecter ms_handle_connect 0x7f52201430c0 2024-03-25T18:48:28.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.456+0000 7f52177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:28.462 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.461+0000 7f52177fe640 10 client.?.objecter ms_handle_connect 0x7f522013f710 2024-03-25T18:48:28.462 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.461+0000 7f52177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:28.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.476+0000 7f52177fe640 10 client.?.objecter ms_handle_connect 0x7f52200a98f0 2024-03-25T18:48:28.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.476+0000 7f52177fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:28.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.536+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2844120120 0 0) 0x7f5220138980 con 0x7f52200a98f0 2024-03-25T18:48:28.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.536+0000 7f52177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:48:28.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.536+0000 7f52177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:28.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.536+0000 7f52177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:28.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.536+0000 7f52177fe640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:48:28.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.536+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5200004eb0 con 0x7f52200a98f0 2024-03-25T18:48:28.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.544+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3596235161 0 0) 0x7f52200b1700 con 0x7f522013f710 2024-03-25T18:48:28.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.544+0000 7f52177fe640 10 monclient(hunting): _negotiate 2024-03-25T18:48:28.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.544+0000 7f52177fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:28.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.544+0000 7f52177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:28.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.544+0000 7f52177fe640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:48:28.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.544+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5220138980 con 0x7f522013f710 2024-03-25T18:48:28.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.556+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4122431653 0 0) 0x7f5200004eb0 con 0x7f52200a98f0 2024-03-25T18:48:28.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.556+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f52200b1700 con 0x7f52200a98f0 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.556+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5210001930 con 0x7f52200a98f0 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.556+0000 7f52177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.556+0000 7f52177fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.556+0000 7f52177fe640 10 monclient(hunting): dump: 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:28.558 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:28.559 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.556+0000 7f52177fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:48:28.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.560+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1391298710 0 0) 0x7f5220138980 con 0x7f522013f710 2024-03-25T18:48:28.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.560+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5200004eb0 con 0x7f522013f710 2024-03-25T18:48:28.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.560+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5208001b40 con 0x7f522013f710 2024-03-25T18:48:28.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.560+0000 7f52177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:28.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.560+0000 7f52177fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:48:28.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.560+0000 7f52177fe640 10 monclient(hunting): dump: 2024-03-25T18:48:28.561 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:28.561 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:28.562 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.562 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.562 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:28.562 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:28.562 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:28.562 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:28.562 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:28.562 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:28.570 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2552825312 0 0) 0x7f52200b1700 con 0x7f52200a98f0 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 >> v1:172.21.15.32:6790/0 conn(0x7f522013f710 legacy=0x7f52201da8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 >> v1:172.21.15.136:6789/0 conn(0x7f52201430c0 legacy=0x7f52201db000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 10 monclient: _finish_hunting 0 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 1 monclient: found mon.a 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52200b1290 con 0x7f52200a98f0 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 10 monclient: _finish_auth 0 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 10 monclient: _check_auth_tickets 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f52177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 10 monclient: _renew_subs 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52201e2eb0 con 0x7f52200a98f0 2024-03-25T18:48:28.571 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:48:28.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 10 monclient: _renew_subs 2024-03-25T18:48:28.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:28.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f52201e3220 con 0x7f52200a98f0 2024-03-25T18:48:28.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.569+0000 7f5225584640 1 librados: init done 2024-03-25T18:48:28.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.570+0000 7f5225584640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:28.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.570+0000 7f5225584640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:28.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.570+0000 7f5225584640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:28.573 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.570+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52200b14e0 con 0x7f52200a98f0 2024-03-25T18:48:28.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.920+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52100035f0 con 0x7f52200a98f0 2024-03-25T18:48:28.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.920+0000 7f52177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:28.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.920+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f52100037e0 con 0x7f52200a98f0 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.920+0000 7f52177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.920+0000 7f52177fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.920+0000 7f52177fe640 10 monclient: dump: 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:28.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.920+0000 7f521dffb640 4 set_mon_vals no callback set 2024-03-25T18:48:28.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.921+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5210003a00 con 0x7f52200a98f0 2024-03-25T18:48:28.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.922+0000 7f5217fff640 10 monclient: get_auth_request con 0x7f520006de90 auth_method 0 2024-03-25T18:48:28.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.922+0000 7f52177fe640 10 client.4653.objecter ms_handle_connect 0x7f520006de90 2024-03-25T18:48:28.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.927+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f5210053440 con 0x7f52200a98f0 2024-03-25T18:48:28.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.927+0000 7f52177fe640 10 client.4653.objecter ms_dispatch 0x7f5220000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:48:28.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.927+0000 7f52177fe640 3 client.4653.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:48:28.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.927+0000 7f52177fe640 3 client.4653.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:48:28.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.927+0000 7f52177fe640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:48:28.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.927+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f521004ef50 con 0x7f52200a98f0 2024-03-25T18:48:28.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.927+0000 7f52177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:28.928 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:28.927+0000 7f52177fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:48:29.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.184+0000 7f5225584640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:29.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.184+0000 7f5225584640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:29.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.184+0000 7f5225584640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:29.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.184+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52200abb30 con 0x7f52200a98f0 2024-03-25T18:48:29.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.208+0000 7f52177fe640 1 -- 172.21.15.32:0/2701765659 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f5210050e80 con 0x7f52200a98f0 2024-03-25T18:48:29.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.208+0000 7f52177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:29.210 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.208+0000 7f52177fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:48:29.210 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:48:29.210 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:48:29.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 10 librados: watch_flush enter 2024-03-25T18:48:29.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 10 librados: watch_flush exit 2024-03-25T18:48:29.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:48:29.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:48:29.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 >> v1:172.21.15.136:6800/1141502808 conn(0x7f520006de90 legacy=0x7f5200070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:29.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 10 monclient: shutdown 2024-03-25T18:48:29.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:29.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 >> v1:172.21.15.32:6789/0 conn(0x7f52200a98f0 legacy=0x7f5220137960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:29.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 shutdown_connections 2024-03-25T18:48:29.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 >> 172.21.15.32:0/2701765659 conn(0x7f52200b7ea0 msgr2=0x7f5220074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:29.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 shutdown_connections 2024-03-25T18:48:29.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 1 -- 172.21.15.32:0/2701765659 wait complete. 2024-03-25T18:48:29.214 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:29.212+0000 7f5225584640 1 librados: shutdown 2024-03-25T18:48:29.225 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:48:34.021 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:46.349 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:34.027 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:46.355 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:34.054 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:09:46.382 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:34.077 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:46.394 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:34.082 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:46.399 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:34.084 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:46.412 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:34.095 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:46.423 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:34.120 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:09:46.437 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:34.131 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:09:46.459 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:36.032 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:48:36.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.148+0000 7f8022102640 10 monclient: get_monmap_and_config 2024-03-25T18:48:36.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.148+0000 7f8022102640 10 monclient: build_initial_monmap 2024-03-25T18:48:36.150 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.148+0000 7f8022102640 10 monclient: monmap: 2024-03-25T18:48:36.150 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:36.150 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:36.150 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:48:36.150094+0000 2024-03-25T18:48:36.150 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:48:36.150094+0000 2024-03-25T18:48:36.151 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:36.151 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:36.151 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:36.151 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:48:36.151 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:48:36.151 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:36.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.149+0000 7f8022102640 1 Processor -- start 2024-03-25T18:48:36.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.149+0000 7f8022102640 1 -- start start 2024-03-25T18:48:36.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.149+0000 7f8022102640 10 monclient: init 2024-03-25T18:48:36.153 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 10 monclient(hunting): picked mon.noname-c con 0x7f801c0a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 10 monclient(hunting): picked mon.noname-a con 0x7f801c13f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 10 monclient(hunting): picked mon.noname-b con 0x7f801c1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c142520 con 0x7f801c13f710 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c0b28a0 con 0x7f801c0a98f0 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c0b2a30 con 0x7f801c1430c0 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:36.154 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.150+0000 7f8022102640 10 monclient(hunting): authenticate will time out at 6622.340088s 2024-03-25T18:48:36.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.162+0000 7f801b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f801c0a98f0 0x7f801c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:54040/0 (socket says 172.21.15.32:54040) 2024-03-25T18:48:36.164 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.162+0000 7f801b7fe640 1 -- 172.21.15.32:0/3284310665 learned_addr learned my addr 172.21.15.32:0/3284310665 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:36.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.180+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3094088 0 0) 0x7f801c0b28a0 con 0x7f801c0a98f0 2024-03-25T18:48:36.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.180+0000 7f801a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:36.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.180+0000 7f801a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:36.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.180+0000 7f801a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:36.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.180+0000 7f801a7fc640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:48:36.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.180+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8004003520 con 0x7f801c0a98f0 2024-03-25T18:48:36.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.187+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3458561066 0 0) 0x7f801c0b2a30 con 0x7f801c1430c0 2024-03-25T18:48:36.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.187+0000 7f801a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:36.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.187+0000 7f801a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:36.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.188+0000 7f801a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:36.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.188+0000 7f801a7fc640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:48:36.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.188+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c0b28a0 con 0x7f801c1430c0 2024-03-25T18:48:36.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973430654 0 0) 0x7f801c142520 con 0x7f801c13f710 2024-03-25T18:48:36.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:36.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:36.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:36.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:48:36.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c0b2a30 con 0x7f801c13f710 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1969273076 0 0) 0x7f8004003520 con 0x7f801c0a98f0 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 >> v1:172.21.15.32:6790/0 conn(0x7f801c1430c0 legacy=0x7f801c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 >> v1:172.21.15.32:6789/0 conn(0x7f801c13f710 legacy=0x7f801c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.193+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c0b44c0 con 0x7f801c0a98f0 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f801a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f801a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f801a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:36.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f800c001690 con 0x7f801c0a98f0 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f801a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f801a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f801a7fc640 10 monclient: dump: 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:36.197 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:36.198 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:36.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f8022102640 5 monclient: authenticate success, global_id 4630 2024-03-25T18:48:36.198 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.194+0000 7f8022102640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:36.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.217+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f800c0017f0 con 0x7f801c0a98f0 2024-03-25T18:48:36.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.217+0000 7f801a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:36.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.217+0000 7f801a7fc640 1 -- 172.21.15.32:0/3284310665 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f800c001c00 con 0x7f801c0a98f0 2024-03-25T18:48:36.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.217+0000 7f8022102640 10 monclient: get_monmap_and_config success 2024-03-25T18:48:36.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.217+0000 7f8022102640 4 set_mon_vals no callback set 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.217+0000 7f801a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.217+0000 7f801a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.217+0000 7f801a7fc640 10 monclient: dump: 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:36.219 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:36.220 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:36.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.218+0000 7f8022102640 10 monclient: shutdown 2024-03-25T18:48:36.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.218+0000 7f8022102640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:36.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.218+0000 7f8022102640 1 -- 172.21.15.32:0/3284310665 >> v1:172.21.15.136:6789/0 conn(0x7f801c0a98f0 legacy=0x7f801c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:36.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.218+0000 7f8022102640 1 -- 172.21.15.32:0/3284310665 shutdown_connections 2024-03-25T18:48:36.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.218+0000 7f8022102640 1 -- 172.21.15.32:0/3284310665 >> 172.21.15.32:0/3284310665 conn(0x7f801c0b7ea0 msgr2=0x7f801c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:36.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.218+0000 7f8022102640 1 -- 172.21.15.32:0/3284310665 shutdown_connections 2024-03-25T18:48:36.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.218+0000 7f8022102640 1 -- 172.21.15.32:0/3284310665 wait complete. 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.219+0000 7f8022102640 10 monclient: build_initial_monmap 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.219+0000 7f8022102640 10 monclient: monmap: 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:36.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.219+0000 7f8022102640 1 librados: starting msgr at 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.219+0000 7f8022102640 1 librados: starting objecter 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.219+0000 7f8022102640 1 Processor -- start 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 1 -- start start 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 1 librados: setting wanted keys 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 1 librados: calling monclient init 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 10 monclient: init 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 10 monclient(hunting): picked mon.noname-c con 0x7f801c0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 10 monclient(hunting): picked mon.noname-b con 0x7f801c13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 10 monclient(hunting): picked mon.noname-a con 0x7f801c1430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c1d7380 con 0x7f801c1430c0 2024-03-25T18:48:36.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c139840 con 0x7f801c13f710 2024-03-25T18:48:36.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f801c1d8200 con 0x7f801c0a98f0 2024-03-25T18:48:36.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:36.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.220+0000 7f8022102640 10 monclient(hunting): authenticate will time out at 6622.410221s 2024-03-25T18:48:36.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.241+0000 7f801b7fe640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f801c0a98f0 0x7f801c1d7af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:49466/0 (socket says 172.21.15.32:49466) 2024-03-25T18:48:36.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.241+0000 7f801b7fe640 1 -- 172.21.15.32:0/2418546949 learned_addr learned my addr 172.21.15.32:0/2418546949 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:36.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.242+0000 7f8003fff640 10 client.?.objecter ms_handle_connect 0x7f801c0a98f0 2024-03-25T18:48:36.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.242+0000 7f8003fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:36.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.244+0000 7f8003fff640 10 client.?.objecter ms_handle_connect 0x7f801c13f710 2024-03-25T18:48:36.245 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.244+0000 7f8003fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:36.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.246+0000 7f8003fff640 10 client.?.objecter ms_handle_connect 0x7f801c1430c0 2024-03-25T18:48:36.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.246+0000 7f8003fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:36.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.247+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2661695327 0 0) 0x7f801c1d8200 con 0x7f801c0a98f0 2024-03-25T18:48:36.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.247+0000 7f8003fff640 10 monclient(hunting): _negotiate 2024-03-25T18:48:36.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.247+0000 7f8003fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:36.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.247+0000 7f8003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:36.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.247+0000 7f8003fff640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:48:36.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.247+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8008004eb0 con 0x7f801c0a98f0 2024-03-25T18:48:36.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.257+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011127621 0 0) 0x7f801c1d7380 con 0x7f801c1430c0 2024-03-25T18:48:36.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.257+0000 7f8003fff640 10 monclient(hunting): _negotiate 2024-03-25T18:48:36.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.257+0000 7f8003fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:36.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.257+0000 7f8003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:36.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.257+0000 7f8003fff640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:48:36.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.257+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f801c1d8200 con 0x7f801c1430c0 2024-03-25T18:48:36.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.262+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2690230947 0 0) 0x7f8008004eb0 con 0x7f801c0a98f0 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.262+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f801c1d7380 con 0x7f801c0a98f0 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.262+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f800c001bc0 con 0x7f801c0a98f0 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.262+0000 7f8003fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.262+0000 7f8003fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.262+0000 7f8003fff640 10 monclient(hunting): dump: 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:36.264 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:36.265 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:36.265 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:36.265 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:36.265 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:36.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.262+0000 7f8003fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:48:36.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.264+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1913400684 0 0) 0x7f801c139840 con 0x7f801c13f710 2024-03-25T18:48:36.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.264+0000 7f8003fff640 10 monclient(hunting): _negotiate 2024-03-25T18:48:36.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.264+0000 7f8003fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:36.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.264+0000 7f8003fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:36.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.264+0000 7f8003fff640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:48:36.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.264+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8008004eb0 con 0x7f801c13f710 2024-03-25T18:48:36.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.272+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3757385288 0 0) 0x7f801c1d8200 con 0x7f801c1430c0 2024-03-25T18:48:36.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.272+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f801c139840 con 0x7f801c1430c0 2024-03-25T18:48:36.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.272+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8014001b50 con 0x7f801c1430c0 2024-03-25T18:48:36.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.272+0000 7f8003fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:36.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.272+0000 7f8003fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:48:36.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.272+0000 7f8003fff640 10 monclient(hunting): dump: 2024-03-25T18:48:36.274 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:36.274 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:36.275 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.275 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.275 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:36.275 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:36.275 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:36.275 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:36.275 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:36.275 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:36.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 18208903 0 0) 0x7f801c139840 con 0x7f801c1430c0 2024-03-25T18:48:36.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 >> v1:172.21.15.32:6790/0 conn(0x7f801c0a98f0 legacy=0x7f801c1d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:36.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 >> v1:172.21.15.136:6789/0 conn(0x7f801c13f710 legacy=0x7f801c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:36.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 10 monclient: _finish_hunting 0 2024-03-25T18:48:36.350 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 1 monclient: found mon.a 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801c1d75d0 con 0x7f801c1430c0 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 10 monclient: _finish_auth 0 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 10 monclient: _check_auth_tickets 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8003fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 10 monclient: _renew_subs 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801c1db6a0 con 0x7f801c1430c0 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 10 client.4659.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 10 monclient: _renew_subs 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:36.351 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f801c137910 con 0x7f801c1430c0 2024-03-25T18:48:36.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.348+0000 7f8022102640 1 librados: init done 2024-03-25T18:48:36.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.349+0000 7f8022102640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:36.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.349+0000 7f8022102640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:36.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.349+0000 7f8022102640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:36.352 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.349+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f801c0627b0 con 0x7f801c1430c0 2024-03-25T18:48:36.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.699+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8014002060 con 0x7f801c1430c0 2024-03-25T18:48:36.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.699+0000 7f8003fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:36.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.699+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f8014003b10 con 0x7f801c1430c0 2024-03-25T18:48:36.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.699+0000 7f8003fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:36.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.699+0000 7f8003fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:48:36.701 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.699+0000 7f8003fff640 10 monclient: dump: 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:36.702 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.700+0000 7f8019ffb640 4 set_mon_vals no callback set 2024-03-25T18:48:36.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.701+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f8014019ae0 con 0x7f801c1430c0 2024-03-25T18:48:36.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.701+0000 7f801b7fe640 10 monclient: get_auth_request con 0x7f800806df00 auth_method 0 2024-03-25T18:48:36.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.701+0000 7f8003fff640 10 client.4659.objecter ms_handle_connect 0x7f800806df00 2024-03-25T18:48:36.708 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.707+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f8014053880 con 0x7f801c1430c0 2024-03-25T18:48:36.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.707+0000 7f8003fff640 10 client.4659.objecter ms_dispatch 0x7f801c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:48:36.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.707+0000 7f8003fff640 3 client.4659.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:48:36.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.707+0000 7f8003fff640 3 client.4659.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:48:36.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.707+0000 7f8003fff640 20 client.4659.objecter dump_active .. 0 homeless 2024-03-25T18:48:36.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.707+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f801404f1c0 con 0x7f801c1430c0 2024-03-25T18:48:36.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.707+0000 7f8003fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:36.709 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.707+0000 7f8003fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:48:36.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.966+0000 7f8022102640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:36.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.966+0000 7f8022102640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:36.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.966+0000 7f8022102640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:48:36.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:36.966+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f801c1d7810 con 0x7f801c1430c0 2024-03-25T18:48:37.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.032+0000 7f8003fff640 1 -- 172.21.15.32:0/2418546949 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f80140512c0 con 0x7f801c1430c0 2024-03-25T18:48:37.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.032+0000 7f8003fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:37.034 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.032+0000 7f8003fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:48:37.034 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:48:37.034 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:48:37.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 10 librados: watch_flush enter 2024-03-25T18:48:37.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 10 librados: watch_flush exit 2024-03-25T18:48:37.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 20 client.4659.objecter shutdown clearing up homeless session... 2024-03-25T18:48:37.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 10 client.4659.objecter successfully canceled tick 2024-03-25T18:48:37.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 >> v1:172.21.15.136:6800/1141502808 conn(0x7f800806df00 legacy=0x7f80080703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:37.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 10 monclient: shutdown 2024-03-25T18:48:37.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:37.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 >> v1:172.21.15.32:6789/0 conn(0x7f801c1430c0 legacy=0x7f801c1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:37.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 shutdown_connections 2024-03-25T18:48:37.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 >> 172.21.15.32:0/2418546949 conn(0x7f801c0b7ea0 msgr2=0x7f801c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:37.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 shutdown_connections 2024-03-25T18:48:37.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 1 -- 172.21.15.32:0/2418546949 wait complete. 2024-03-25T18:48:37.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:37.035+0000 7f8022102640 1 librados: shutdown 2024-03-25T18:48:37.049 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:48:43.857 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.972+0000 7f491facf640 10 monclient: get_monmap_and_config 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.972+0000 7f491facf640 10 monclient: build_initial_monmap 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.972+0000 7f491facf640 10 monclient: monmap: 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:48:43.972538+0000 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:48:43.972538+0000 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:43.972 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:43.973 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:43.973 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:48:43.973 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:48:43.973 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:43.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.972+0000 7f491facf640 1 Processor -- start 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.972+0000 7f491facf640 1 -- start start 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.972+0000 7f491facf640 10 monclient: init 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 10 monclient(hunting): picked mon.noname-a con 0x7f49180a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 10 monclient(hunting): picked mon.noname-b con 0x7f491813f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 10 monclient(hunting): picked mon.noname-c con 0x7f49181430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4918142520 con 0x7f49180a98f0 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180b28a0 con 0x7f49181430c0 2024-03-25T18:48:43.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180b2a30 con 0x7f491813f710 2024-03-25T18:48:43.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:43.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.973+0000 7f491facf640 10 monclient(hunting): authenticate will time out at 6630.162224s 2024-03-25T18:48:43.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.975+0000 7f491d06a640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f491813f710 0x7f49181429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:35234/0 (socket says 172.21.15.32:35234) 2024-03-25T18:48:43.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:43.975+0000 7f491d06a640 1 -- 172.21.15.32:0/900442444 learned_addr learned my addr 172.21.15.32:0/900442444 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:44.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.004+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364895522 0 0) 0x7f49180b2a30 con 0x7f491813f710 2024-03-25T18:48:44.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.004+0000 7f491c869640 10 monclient(hunting): _negotiate 2024-03-25T18:48:44.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.004+0000 7f491c869640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:44.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.004+0000 7f491c869640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:44.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.004+0000 7f491c869640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:48:44.004 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.004+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904003520 con 0x7f491813f710 2024-03-25T18:48:44.007 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.007+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 847726622 0 0) 0x7f4918142520 con 0x7f49180a98f0 2024-03-25T18:48:44.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.007+0000 7f491c869640 10 monclient(hunting): _negotiate 2024-03-25T18:48:44.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.007+0000 7f491c869640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:44.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.007+0000 7f491c869640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:44.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.007+0000 7f491c869640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:48:44.008 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.007+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49180b2a30 con 0x7f49180a98f0 2024-03-25T18:48:44.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1610359338 0 0) 0x7f49180b2a30 con 0x7f49180a98f0 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 >> v1:172.21.15.32:6790/0 conn(0x7f491813f710 legacy=0x7f49181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 >> v1:172.21.15.136:6789/0 conn(0x7f49181430c0 legacy=0x7f491814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 10 monclient: _finish_hunting 0 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 1 monclient: found mon.noname-a 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180b44c0 con 0x7f49180a98f0 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 10 monclient: _finish_auth 0 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 10 monclient: _check_auth_tickets 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4914001670 con 0x7f49180a98f0 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491c869640 10 monclient: dump: 2024-03-25T18:48:44.014 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491facf640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:48:44.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.013+0000 7f491facf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:44.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.035+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49140017d0 con 0x7f49180a98f0 2024-03-25T18:48:44.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.035+0000 7f491c869640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:44.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.035+0000 7f491c869640 1 -- 172.21.15.32:0/900442444 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4914001ba0 con 0x7f49180a98f0 2024-03-25T18:48:44.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.035+0000 7f491c869640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:44.035 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.035+0000 7f491c869640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.035+0000 7f491c869640 10 monclient: dump: 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.035+0000 7f491facf640 10 monclient: get_monmap_and_config success 2024-03-25T18:48:44.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.035+0000 7f491facf640 4 set_mon_vals no callback set 2024-03-25T18:48:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.036+0000 7f491facf640 10 monclient: shutdown 2024-03-25T18:48:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.036+0000 7f491facf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.036+0000 7f491facf640 1 -- 172.21.15.32:0/900442444 >> v1:172.21.15.32:6789/0 conn(0x7f49180a98f0 legacy=0x7f49180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.036+0000 7f491facf640 1 -- 172.21.15.32:0/900442444 shutdown_connections 2024-03-25T18:48:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.036+0000 7f491facf640 1 -- 172.21.15.32:0/900442444 >> 172.21.15.32:0/900442444 conn(0x7f49180b7ea0 msgr2=0x7f49180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.036+0000 7f491facf640 1 -- 172.21.15.32:0/900442444 shutdown_connections 2024-03-25T18:48:44.037 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.036+0000 7f491facf640 1 -- 172.21.15.32:0/900442444 wait complete. 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.037+0000 7f491facf640 10 monclient: build_initial_monmap 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.037+0000 7f491facf640 10 monclient: monmap: 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:48:44.038 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:44.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.038+0000 7f491facf640 1 librados: starting msgr at 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.038+0000 7f491facf640 1 librados: starting objecter 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.038+0000 7f491facf640 1 Processor -- start 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.038+0000 7f491facf640 1 -- start start 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.038+0000 7f491facf640 1 librados: setting wanted keys 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.038+0000 7f491facf640 1 librados: calling monclient init 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.038+0000 7f491facf640 10 monclient: init 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 10 monclient(hunting): picked mon.noname-b con 0x7f49180a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 10 monclient(hunting): picked mon.noname-a con 0x7f491813f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 10 monclient(hunting): picked mon.noname-c con 0x7f49181430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49181daa20 con 0x7f491813f710 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49181371e0 con 0x7f49180a98f0 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49181374c0 con 0x7f49181430c0 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:44.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.039+0000 7f491facf640 10 monclient(hunting): authenticate will time out at 6630.227837s 2024-03-25T18:48:44.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.059+0000 7f491d06a640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f491813f710 0x7f49181daf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:42646/0 (socket says 172.21.15.32:42646) 2024-03-25T18:48:44.060 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.059+0000 7f491d06a640 1 -- 172.21.15.32:0/2452294188 learned_addr learned my addr 172.21.15.32:0/2452294188 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:44.060 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.060+0000 7f490e7fc640 10 client.?.objecter ms_handle_connect 0x7f491813f710 2024-03-25T18:48:44.060 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.060+0000 7f490e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:44.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.069+0000 7f490e7fc640 10 client.?.objecter ms_handle_connect 0x7f49180a98f0 2024-03-25T18:48:44.069 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.069+0000 7f490e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:44.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.070+0000 7f490e7fc640 10 client.?.objecter ms_handle_connect 0x7f49181430c0 2024-03-25T18:48:44.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.070+0000 7f490e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:44.070 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.070+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2453721642 0 0) 0x7f49181daa20 con 0x7f491813f710 2024-03-25T18:48:44.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.070+0000 7f490e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:44.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.070+0000 7f490e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:44.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.070+0000 7f490e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:44.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.070+0000 7f490e7fc640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:48:44.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.071+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48fc004eb0 con 0x7f491813f710 2024-03-25T18:48:44.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299587050 0 0) 0x7f49181371e0 con 0x7f49180a98f0 2024-03-25T18:48:44.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:44.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:44.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:44.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:48:44.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49181daa20 con 0x7f49180a98f0 2024-03-25T18:48:44.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2231474926 0 0) 0x7f49181374c0 con 0x7f49181430c0 2024-03-25T18:48:44.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:44.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:44.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:44.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:48:44.076 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.075+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49181371e0 con 0x7f49181430c0 2024-03-25T18:48:44.090 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.090+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3706339316 0 0) 0x7f49181daa20 con 0x7f49180a98f0 2024-03-25T18:48:44.090 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.090+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49181374c0 con 0x7f49180a98f0 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.090+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4914001ae0 con 0x7f49180a98f0 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.090+0000 7f490e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.090+0000 7f490e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.090+0000 7f490e7fc640 10 monclient(hunting): dump: 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:44.091 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.090+0000 7f490e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:48:44.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.093+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2409569476 0 0) 0x7f48fc004eb0 con 0x7f491813f710 2024-03-25T18:48:44.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.093+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f49181daa20 con 0x7f491813f710 2024-03-25T18:48:44.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.093+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4908001990 con 0x7f491813f710 2024-03-25T18:48:44.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.093+0000 7f490e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.093+0000 7f490e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.093+0000 7f490e7fc640 10 monclient(hunting): dump: 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:44.094 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.095+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1496949273 0 0) 0x7f49181371e0 con 0x7f49181430c0 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.096+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48fc004eb0 con 0x7f49181430c0 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.096+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4910001a60 con 0x7f49181430c0 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.096+0000 7f490e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.096+0000 7f490e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.096+0000 7f490e7fc640 10 monclient(hunting): dump: 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:44.096 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:44.097 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:44.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3760139867 0 0) 0x7f49181374c0 con 0x7f49180a98f0 2024-03-25T18:48:44.111 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 >> v1:172.21.15.32:6790/0 conn(0x7f49181430c0 legacy=0x7f49181e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 >> v1:172.21.15.32:6789/0 conn(0x7f491813f710 legacy=0x7f49181daf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 1 monclient: found mon.b 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491813c040 con 0x7f49180a98f0 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f490e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 10 monclient: _renew_subs 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49181359f0 con 0x7f49180a98f0 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 10 client.4639.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:48:44.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 10 monclient: _renew_subs 2024-03-25T18:48:44.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:44.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f49181d75d0 con 0x7f49180a98f0 2024-03-25T18:48:44.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.111+0000 7f491facf640 1 librados: init done 2024-03-25T18:48:44.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.112+0000 7f491facf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:44.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.112+0000 7f491facf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:44.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.112+0000 7f491facf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:44.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.112+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49180627b0 con 0x7f49180a98f0 2024-03-25T18:48:44.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.402+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49140038a0 con 0x7f49180a98f0 2024-03-25T18:48:44.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.402+0000 7f490e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:44.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.402+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4914003bb0 con 0x7f49180a98f0 2024-03-25T18:48:44.402 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.402+0000 7f490e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.402+0000 7f490e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.402+0000 7f490e7fc640 10 monclient: dump: 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:44.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.402+0000 7f490ffff640 4 set_mon_vals no callback set 2024-03-25T18:48:44.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.403+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f4914019b80 con 0x7f49180a98f0 2024-03-25T18:48:44.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.403+0000 7f491d06a640 10 monclient: get_auth_request con 0x7f48fc06e0b0 auth_method 0 2024-03-25T18:48:44.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.404+0000 7f490e7fc640 10 client.4639.objecter ms_handle_connect 0x7f48fc06e0b0 2024-03-25T18:48:44.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.406+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f4914053620 con 0x7f49180a98f0 2024-03-25T18:48:44.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.406+0000 7f490e7fc640 10 client.4639.objecter ms_dispatch 0x7f4918000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:48:44.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.406+0000 7f490e7fc640 3 client.4639.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:48:44.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.406+0000 7f490e7fc640 3 client.4639.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:48:44.407 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.406+0000 7f490e7fc640 20 client.4639.objecter dump_active .. 0 homeless 2024-03-25T18:48:44.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.410+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f491404f130 con 0x7f49180a98f0 2024-03-25T18:48:44.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.410+0000 7f490e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:44.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.410+0000 7f490e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:48:44.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.615+0000 7f491facf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:44.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.615+0000 7f491facf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:44.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.615+0000 7f491facf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:44.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.615+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49180008d0 con 0x7f49180a98f0 2024-03-25T18:48:44.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.651+0000 7f490e7fc640 1 -- 172.21.15.32:0/2452294188 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f4914051060 con 0x7f49180a98f0 2024-03-25T18:48:44.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.651+0000 7f490e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:44.651 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.651+0000 7f490e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:48:44.652 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:48:44.652 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:48:44.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 10 librados: watch_flush enter 2024-03-25T18:48:44.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 10 librados: watch_flush exit 2024-03-25T18:48:44.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 20 client.4639.objecter shutdown clearing up homeless session... 2024-03-25T18:48:44.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 10 client.4639.objecter successfully canceled tick 2024-03-25T18:48:44.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 >> v1:172.21.15.136:6800/1141502808 conn(0x7f48fc06e0b0 legacy=0x7f48fc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:44.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 10 monclient: shutdown 2024-03-25T18:48:44.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:44.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 >> v1:172.21.15.136:6789/0 conn(0x7f49180a98f0 legacy=0x7f49181352e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:44.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 shutdown_connections 2024-03-25T18:48:44.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 >> 172.21.15.32:0/2452294188 conn(0x7f49180b7ea0 msgr2=0x7f4918074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:44.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.653+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 shutdown_connections 2024-03-25T18:48:44.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.654+0000 7f491facf640 1 -- 172.21.15.32:0/2452294188 wait complete. 2024-03-25T18:48:44.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:44.654+0000 7f491facf640 1 librados: shutdown 2024-03-25T18:48:44.666 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:48:51.474 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:48:51.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.590+0000 7f204fd8d640 10 monclient: get_monmap_and_config 2024-03-25T18:48:51.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.590+0000 7f204fd8d640 10 monclient: build_initial_monmap 2024-03-25T18:48:51.591 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.590+0000 7f204fd8d640 10 monclient: monmap: 2024-03-25T18:48:51.591 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:51.591 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:51.591 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:48:51.591360+0000 2024-03-25T18:48:51.591 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:48:51.591360+0000 2024-03-25T18:48:51.592 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:51.592 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:51.592 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:51.592 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:48:51.592 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:48:51.592 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:51.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.591+0000 7f204fd8d640 1 Processor -- start 2024-03-25T18:48:51.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.592+0000 7f204fd8d640 1 -- start start 2024-03-25T18:48:51.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.592+0000 7f204fd8d640 10 monclient: init 2024-03-25T18:48:51.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:51.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:48:51.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 10 monclient(hunting): picked mon.noname-b con 0x7f204806adf0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:51.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 10 monclient(hunting): picked mon.noname-c con 0x7f20480727c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:51.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 10 monclient(hunting): picked mon.noname-a con 0x7f204806f3b0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:51.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20480704c0 con 0x7f204806f3b0 2024-03-25T18:48:51.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2048070730 con 0x7f20480727c0 2024-03-25T18:48:51.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20480b1060 con 0x7f204806adf0 2024-03-25T18:48:51.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:51.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.593+0000 7f204fd8d640 10 monclient(hunting): authenticate will time out at 6637.781686s 2024-03-25T18:48:51.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.601+0000 7f204d328640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f20480727c0 0x7f2048146760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:45900/0 (socket says 172.21.15.32:45900) 2024-03-25T18:48:51.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.601+0000 7f204d328640 1 -- 172.21.15.32:0/3429094416 learned_addr learned my addr 172.21.15.32:0/3429094416 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:51.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.607+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3839889038 0 0) 0x7f2048070730 con 0x7f20480727c0 2024-03-25T18:48:51.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.607+0000 7f204cb27640 10 monclient(hunting): _negotiate 2024-03-25T18:48:51.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.607+0000 7f204cb27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:51.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.607+0000 7f204cb27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:51.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.607+0000 7f204cb27640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:48:51.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.607+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2034003520 con 0x7f20480727c0 2024-03-25T18:48:51.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.612+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1095516148 0 0) 0x7f20480b1060 con 0x7f204806adf0 2024-03-25T18:48:51.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.612+0000 7f204cb27640 10 monclient(hunting): _negotiate 2024-03-25T18:48:51.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.612+0000 7f204cb27640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:51.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.612+0000 7f204cb27640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:51.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.612+0000 7f204cb27640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:48:51.612 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.612+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2048070730 con 0x7f204806adf0 2024-03-25T18:48:51.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.612+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1549062853 0 0) 0x7f2034003520 con 0x7f20480727c0 2024-03-25T18:48:51.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 >> v1:172.21.15.32:6790/0 conn(0x7f204806adf0 legacy=0x7f20480720b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:51.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 >> v1:172.21.15.32:6789/0 conn(0x7f204806f3b0 legacy=0x7f2048152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:51.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 10 monclient: _finish_hunting 0 2024-03-25T18:48:51.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 1 monclient: found mon.noname-c 2024-03-25T18:48:51.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:51.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:48:51.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20480b0b40 con 0x7f20480727c0 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 10 monclient: _finish_auth 0 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 10 monclient: _check_auth_tickets 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2038001690 con 0x7f20480727c0 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204fd8d640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204fd8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204cb27640 10 monclient: dump: 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:51.615 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:51.616 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.613+0000 7f204fd8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:51.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.633+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20380017f0 con 0x7f20480727c0 2024-03-25T18:48:51.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.633+0000 7f204cb27640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:51.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.633+0000 7f204cb27640 1 -- 172.21.15.32:0/3429094416 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2038001b80 con 0x7f20480727c0 2024-03-25T18:48:51.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.633+0000 7f204cb27640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:51.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.633+0000 7f204cb27640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:51.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.633+0000 7f204cb27640 10 monclient: dump: 2024-03-25T18:48:51.634 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:51.634 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.633+0000 7f204fd8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:48:51.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.633+0000 7f204fd8d640 4 set_mon_vals no callback set 2024-03-25T18:48:51.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.634+0000 7f204fd8d640 10 monclient: shutdown 2024-03-25T18:48:51.637 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.634+0000 7f204fd8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.634+0000 7f204fd8d640 1 -- 172.21.15.32:0/3429094416 >> v1:172.21.15.136:6789/0 conn(0x7f20480727c0 legacy=0x7f2048146760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.634+0000 7f204fd8d640 1 -- 172.21.15.32:0/3429094416 shutdown_connections 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.634+0000 7f204fd8d640 1 -- 172.21.15.32:0/3429094416 >> 172.21.15.32:0/3429094416 conn(0x7f20480b4a80 msgr2=0x7f20480b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.634+0000 7f204fd8d640 1 -- 172.21.15.32:0/3429094416 shutdown_connections 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.635+0000 7f204fd8d640 1 -- 172.21.15.32:0/3429094416 wait complete. 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.635+0000 7f204fd8d640 10 monclient: build_initial_monmap 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.635+0000 7f204fd8d640 10 monclient: monmap: 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:48:51.638 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.635+0000 7f204fd8d640 1 librados: starting msgr at 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.635+0000 7f204fd8d640 1 librados: starting objecter 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 1 Processor -- start 2024-03-25T18:48:51.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 1 -- start start 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 1 librados: setting wanted keys 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 1 librados: calling monclient init 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 10 monclient: init 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 10 monclient(hunting): picked mon.noname-c con 0x7f204806adf0 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 10 monclient(hunting): picked mon.noname-b con 0x7f204806f3b0 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 10 monclient(hunting): picked mon.noname-a con 0x7f20480727c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20481d2d10 con 0x7f20480727c0 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20481d0c90 con 0x7f204806f3b0 2024-03-25T18:48:51.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20481dc180 con 0x7f204806adf0 2024-03-25T18:48:51.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:51.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.636+0000 7f204fd8d640 10 monclient(hunting): authenticate will time out at 6637.825308s 2024-03-25T18:48:51.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.647+0000 7f204e32a640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f20480727c0 0x7f20481e79c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:36534/0 (socket says 172.21.15.32:36534) 2024-03-25T18:48:51.649 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.647+0000 7f204e32a640 1 -- 172.21.15.32:0/4185841088 learned_addr learned my addr 172.21.15.32:0/4185841088 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:51.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.649+0000 7f203e7fc640 10 client.?.objecter ms_handle_connect 0x7f20480727c0 2024-03-25T18:48:51.650 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.649+0000 7f203e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:51.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.655+0000 7f203e7fc640 10 client.?.objecter ms_handle_connect 0x7f204806f3b0 2024-03-25T18:48:51.655 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.655+0000 7f203e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:51.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.662+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 865970716 0 0) 0x7f20481d2d10 con 0x7f20480727c0 2024-03-25T18:48:51.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.662+0000 7f203e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:51.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.662+0000 7f203e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:51.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.662+0000 7f203e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:51.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.662+0000 7f203e7fc640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:48:51.663 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.662+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f202c004eb0 con 0x7f20480727c0 2024-03-25T18:48:51.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.664+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1220974677 0 0) 0x7f20481d0c90 con 0x7f204806f3b0 2024-03-25T18:48:51.664 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.664+0000 7f203e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:51.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.664+0000 7f203e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:51.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.664+0000 7f203e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:51.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.664+0000 7f203e7fc640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:48:51.665 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.664+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20481d2d10 con 0x7f204806f3b0 2024-03-25T18:48:51.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.669+0000 7f203e7fc640 10 client.?.objecter ms_handle_connect 0x7f204806adf0 2024-03-25T18:48:51.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.669+0000 7f203e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:51.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.682+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3419041308 0 0) 0x7f202c004eb0 con 0x7f20480727c0 2024-03-25T18:48:51.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.682+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20481d0c90 con 0x7f20480727c0 2024-03-25T18:48:51.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.682+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2040001d30 con 0x7f20480727c0 2024-03-25T18:48:51.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.682+0000 7f203e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:51.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.682+0000 7f203e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.682+0000 7f203e7fc640 10 monclient(hunting): dump: 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:51.684 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.682+0000 7f203e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:48:51.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.683+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622414364 0 0) 0x7f20481dc180 con 0x7f204806adf0 2024-03-25T18:48:51.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.683+0000 7f203e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:51.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.683+0000 7f203e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:51.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.683+0000 7f203e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:51.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.683+0000 7f203e7fc640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:48:51.685 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.683+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f202c004eb0 con 0x7f204806adf0 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.687+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3305951919 0 0) 0x7f20481d2d10 con 0x7f204806f3b0 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.687+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f20481dc180 con 0x7f204806f3b0 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.687+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2038001b00 con 0x7f204806f3b0 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.687+0000 7f203e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.687+0000 7f203e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.687+0000 7f203e7fc640 10 monclient(hunting): dump: 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:51.688 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.689 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.689 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:51.689 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:51.689 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:51.689 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:51.689 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:51.689 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:51.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3681142581 0 0) 0x7f20481dc180 con 0x7f204806f3b0 2024-03-25T18:48:51.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 >> v1:172.21.15.32:6790/0 conn(0x7f204806adf0 legacy=0x7f20481dc630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:51.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 >> v1:172.21.15.32:6789/0 conn(0x7f20480727c0 legacy=0x7f20481e79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:51.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:48:51.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 1 monclient: found mon.b 2024-03-25T18:48:51.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:51.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20481dcd40 con 0x7f204806f3b0 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.703+0000 7f203e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 10 monclient: _renew_subs 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20481dcf30 con 0x7f204806f3b0 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 10 monclient: _renew_subs 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f20481d0800 con 0x7f204806f3b0 2024-03-25T18:48:51.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.704+0000 7f204fd8d640 1 librados: init done 2024-03-25T18:48:51.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.705+0000 7f204fd8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:51.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.705+0000 7f204fd8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:51.706 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.705+0000 7f204fd8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:51.707 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.705+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20480627b0 con 0x7f204806f3b0 2024-03-25T18:48:51.981 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.981+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2038001c60 con 0x7f204806f3b0 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.981+0000 7f203e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.981+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2038004c30 con 0x7f204806f3b0 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.981+0000 7f203e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.981+0000 7f203e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.981+0000 7f203ffff640 4 set_mon_vals no callback set 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.981+0000 7f203e7fc640 10 monclient: dump: 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:51.982 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:51.983 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:51.983 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:51.983 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:51.983 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:51.983 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:51.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.982+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f203801ac00 con 0x7f204806f3b0 2024-03-25T18:48:51.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.983+0000 7f204db29640 10 monclient: get_auth_request con 0x7f202c06df70 auth_method 0 2024-03-25T18:48:51.983 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.983+0000 7f203e7fc640 10 client.4645.objecter ms_handle_connect 0x7f202c06df70 2024-03-25T18:48:51.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.988+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f20380547a0 con 0x7f204806f3b0 2024-03-25T18:48:51.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.988+0000 7f203e7fc640 10 client.4645.objecter ms_dispatch 0x7f2048000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:48:51.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.988+0000 7f203e7fc640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:48:51.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.988+0000 7f203e7fc640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:48:51.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.988+0000 7f203e7fc640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T18:48:51.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.988+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20380502b0 con 0x7f204806f3b0 2024-03-25T18:48:51.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.989+0000 7f203e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:51.989 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:51.989+0000 7f203e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:48:52.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.250+0000 7f204fd8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:52.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.250+0000 7f204fd8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:52.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.250+0000 7f204fd8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:52.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.250+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2048070170 con 0x7f204806f3b0 2024-03-25T18:48:52.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.282+0000 7f203e7fc640 1 -- 172.21.15.32:0/4185841088 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f20380521e0 con 0x7f204806f3b0 2024-03-25T18:48:52.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.282+0000 7f203e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:52.283 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.282+0000 7f203e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:48:52.283 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:48:52.283 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:48:52.285 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.284+0000 7f204fd8d640 10 librados: watch_flush enter 2024-03-25T18:48:52.285 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.284+0000 7f204fd8d640 10 librados: watch_flush exit 2024-03-25T18:48:52.285 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.284+0000 7f204fd8d640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T18:48:52.285 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.284+0000 7f204fd8d640 10 client.4645.objecter successfully canceled tick 2024-03-25T18:48:52.285 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 >> v1:172.21.15.136:6800/1141502808 conn(0x7f202c06df70 legacy=0x7f202c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:52.285 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 10 monclient: shutdown 2024-03-25T18:48:52.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:52.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 >> v1:172.21.15.136:6789/0 conn(0x7f204806f3b0 legacy=0x7f20481d1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:52.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 shutdown_connections 2024-03-25T18:48:52.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 >> 172.21.15.32:0/4185841088 conn(0x7f20480b4a80 msgr2=0x7f20480b3f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:52.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 shutdown_connections 2024-03-25T18:48:52.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 1 -- 172.21.15.32:0/4185841088 wait complete. 2024-03-25T18:48:52.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:52.285+0000 7f204fd8d640 1 librados: shutdown 2024-03-25T18:48:52.296 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:48:54.068 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:06.396 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:54.119 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:06.447 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:54.151 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:06.479 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:54.165 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:06.482 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:54.169 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:06.486 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:54.196 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:06.524 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:54.202 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:06.519 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:54.204 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:06.532 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:54.237 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:06.565 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:48:59.103 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.215+0000 7f2771c4a640 10 monclient: get_monmap_and_config 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.215+0000 7f2771c4a640 10 monclient: build_initial_monmap 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.215+0000 7f2771c4a640 10 monclient: monmap: 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:48:59.215754+0000 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:48:59.215754+0000 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:59.215 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:59.216 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:48:59.216 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:48:59.216 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:59.216 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.216+0000 7f2771c4a640 1 Processor -- start 2024-03-25T18:48:59.216 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.216+0000 7f2771c4a640 1 -- start start 2024-03-25T18:48:59.216 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.216+0000 7f2771c4a640 10 monclient: init 2024-03-25T18:48:59.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:59.217 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:48:59.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f276c0b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:59.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f276c0b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:59.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f276c14be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:59.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276c0af940 con 0x7f276c0b4d60 2024-03-25T18:48:59.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276c0b0a70 con 0x7f276c14be90 2024-03-25T18:48:59.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276c0b0d50 con 0x7f276c0b1730 2024-03-25T18:48:59.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:59.218 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.217+0000 7f2771c4a640 10 monclient(hunting): authenticate will time out at 6645.406199s 2024-03-25T18:48:59.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.233+0000 7f276bfff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f276c14be90 0x7f276c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43978/0 (socket says 172.21.15.32:43978) 2024-03-25T18:48:59.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.233+0000 7f276bfff640 1 -- 172.21.15.32:0/2703444012 learned_addr learned my addr 172.21.15.32:0/2703444012 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:59.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.240+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321853164 0 0) 0x7f276c0b0a70 con 0x7f276c14be90 2024-03-25T18:48:59.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.240+0000 7f276a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:48:59.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.240+0000 7f276a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:59.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.240+0000 7f276a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:59.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.240+0000 7f276a7fc640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:48:59.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.240+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2754003520 con 0x7f276c14be90 2024-03-25T18:48:59.247 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 32965028 0 0) 0x7f2754003520 con 0x7f276c14be90 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 >> v1:172.21.15.32:6790/0 conn(0x7f276c0b1730 legacy=0x7f276c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 >> v1:172.21.15.32:6789/0 conn(0x7f276c0b4d60 legacy=0x7f276c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c0b1030 con 0x7f276c14be90 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2764001a10 con 0x7f276c14be90 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f2771c4a640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f2771c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:59.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:59.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:48:59.249 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f276a7fc640 10 monclient: dump: 2024-03-25T18:48:59.249 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:59.249 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:59.249 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:59.249 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:59.249 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:59.250 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:59.250 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:59.250 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:59.250 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:59.250 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:59.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.247+0000 7f2771c4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:48:59.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.254+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2764001b70 con 0x7f276c14be90 2024-03-25T18:48:59.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.254+0000 7f276a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:59.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.254+0000 7f276a7fc640 1 -- 172.21.15.32:0/2703444012 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2764001f80 con 0x7f276c14be90 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.254+0000 7f276a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.254+0000 7f276a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.254+0000 7f276a7fc640 10 monclient: dump: 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:59.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.254+0000 7f2771c4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:48:59.256 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.254+0000 7f2771c4a640 4 set_mon_vals no callback set 2024-03-25T18:48:59.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.255+0000 7f2771c4a640 10 monclient: shutdown 2024-03-25T18:48:59.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.255+0000 7f2771c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:59.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.255+0000 7f2771c4a640 1 -- 172.21.15.32:0/2703444012 >> v1:172.21.15.136:6789/0 conn(0x7f276c14be90 legacy=0x7f276c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:59.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.255+0000 7f2771c4a640 1 -- 172.21.15.32:0/2703444012 shutdown_connections 2024-03-25T18:48:59.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.255+0000 7f2771c4a640 1 -- 172.21.15.32:0/2703444012 >> 172.21.15.32:0/2703444012 conn(0x7f276c0617a0 msgr2=0x7f276c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:59.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.255+0000 7f2771c4a640 1 -- 172.21.15.32:0/2703444012 shutdown_connections 2024-03-25T18:48:59.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.255+0000 7f2771c4a640 1 -- 172.21.15.32:0/2703444012 wait complete. 2024-03-25T18:48:59.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 10 monclient: build_initial_monmap 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 10 monclient: monmap: 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 1 librados: starting msgr at 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 1 librados: starting objecter 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 1 Processor -- start 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 1 -- start start 2024-03-25T18:48:59.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 1 librados: setting wanted keys 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 1 librados: calling monclient init 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.256+0000 7f2771c4a640 10 monclient: init 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 10 monclient(hunting): picked mon.noname-b con 0x7f276c0b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 10 monclient(hunting): picked mon.noname-c con 0x7f276c0b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 10 monclient(hunting): picked mon.noname-a con 0x7f276c14be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276c135ee0 con 0x7f276c14be90 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276c135b00 con 0x7f276c0b1730 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f276c135c90 con 0x7f276c0b4d60 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:48:59.260 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.257+0000 7f2771c4a640 10 monclient(hunting): authenticate will time out at 6645.445933s 2024-03-25T18:48:59.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.260+0000 7f276b7fe640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f276c0b1730 0x7f276c12fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:44010/0 (socket says 172.21.15.32:44010) 2024-03-25T18:48:59.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.260+0000 7f276b7fe640 1 -- 172.21.15.32:0/980707052 learned_addr learned my addr 172.21.15.32:0/980707052 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:48:59.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.261+0000 7f2753fff640 10 client.?.objecter ms_handle_connect 0x7f276c0b1730 2024-03-25T18:48:59.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.261+0000 7f2753fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:59.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.270+0000 7f2753fff640 10 client.?.objecter ms_handle_connect 0x7f276c14be90 2024-03-25T18:48:59.270 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.270+0000 7f2753fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:59.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.272+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617957349 0 0) 0x7f276c135b00 con 0x7f276c0b1730 2024-03-25T18:48:59.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.272+0000 7f2753fff640 10 monclient(hunting): _negotiate 2024-03-25T18:48:59.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.272+0000 7f2753fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:59.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.272+0000 7f2753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:59.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.272+0000 7f2753fff640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:48:59.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.272+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2758004eb0 con 0x7f276c0b1730 2024-03-25T18:48:59.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.273+0000 7f2753fff640 10 client.?.objecter ms_handle_connect 0x7f276c0b4d60 2024-03-25T18:48:59.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.273+0000 7f2753fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:48:59.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.286+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3973521738 0 0) 0x7f276c135ee0 con 0x7f276c14be90 2024-03-25T18:48:59.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.286+0000 7f2753fff640 10 monclient(hunting): _negotiate 2024-03-25T18:48:59.287 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.286+0000 7f2753fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:59.287 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.286+0000 7f2753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:59.287 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.286+0000 7f2753fff640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:48:59.287 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.286+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f276c135b00 con 0x7f276c14be90 2024-03-25T18:48:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.289+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1242765263 0 0) 0x7f276c135c90 con 0x7f276c0b4d60 2024-03-25T18:48:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 10 monclient(hunting): _negotiate 2024-03-25T18:48:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:48:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:48:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:48:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f276c135ee0 con 0x7f276c0b4d60 2024-03-25T18:48:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3846418108 0 0) 0x7f2758004eb0 con 0x7f276c0b1730 2024-03-25T18:48:59.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f276c135c90 con 0x7f276c0b1730 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f275c001a30 con 0x7f276c0b1730 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 10 monclient(hunting): dump: 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:59.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.290+0000 7f2753fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:48:59.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2799300510 0 0) 0x7f276c135c90 con 0x7f276c0b1730 2024-03-25T18:48:59.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 >> v1:172.21.15.32:6790/0 conn(0x7f276c0b4d60 legacy=0x7f276c1d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:59.303 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 >> v1:172.21.15.32:6789/0 conn(0x7f276c14be90 legacy=0x7f276c1d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 10 monclient: _finish_hunting 0 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 1 monclient: found mon.b 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f276c136960 con 0x7f276c0b1730 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 10 monclient: _finish_auth 0 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 10 monclient: _check_auth_tickets 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2753fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 10 monclient: _renew_subs 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f276c130470 con 0x7f276c0b1730 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 10 client.4651.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:48:59.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 10 monclient: _renew_subs 2024-03-25T18:48:59.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:59.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f276c13e870 con 0x7f276c0b1730 2024-03-25T18:48:59.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.303+0000 7f2771c4a640 1 librados: init done 2024-03-25T18:48:59.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.304+0000 7f2771c4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:59.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.304+0000 7f2771c4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:59.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.304+0000 7f2771c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:59.305 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.304+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f276c130810 con 0x7f276c0b1730 2024-03-25T18:48:59.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.526+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f275c001c70 con 0x7f276c0b1730 2024-03-25T18:48:59.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.526+0000 7f2753fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:48:59.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.526+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f275c0038c0 con 0x7f276c0b1730 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.526+0000 7f2753fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.526+0000 7f2753fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.526+0000 7f2753fff640 10 monclient: dump: 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:48:59.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.526+0000 7f2769ffb640 4 set_mon_vals no callback set 2024-03-25T18:48:59.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.589+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f275c003a20 con 0x7f276c0b1730 2024-03-25T18:48:59.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.590+0000 7f276affd640 10 monclient: get_auth_request con 0x7f275806ddf0 auth_method 0 2024-03-25T18:48:59.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.590+0000 7f2753fff640 10 client.4651.objecter ms_handle_connect 0x7f275806ddf0 2024-03-25T18:48:59.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.593+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f275c054730 con 0x7f276c0b1730 2024-03-25T18:48:59.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.593+0000 7f2753fff640 10 client.4651.objecter ms_dispatch 0x7f276c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:48:59.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.593+0000 7f2753fff640 3 client.4651.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:48:59.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.593+0000 7f2753fff640 3 client.4651.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:48:59.593 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.593+0000 7f2753fff640 20 client.4651.objecter dump_active .. 0 homeless 2024-03-25T18:48:59.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.596+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f275c050070 con 0x7f276c0b1730 2024-03-25T18:48:59.596 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.596+0000 7f2753fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:48:59.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.596+0000 7f2753fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:48:59.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.821+0000 7f2771c4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:59.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.821+0000 7f2771c4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:59.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.821+0000 7f2771c4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:48:59.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.821+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f276c130600 con 0x7f276c0b1730 2024-03-25T18:48:59.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.861+0000 7f2753fff640 1 -- 172.21.15.32:0/980707052 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f275c052170 con 0x7f276c0b1730 2024-03-25T18:48:59.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.861+0000 7f2753fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:48:59.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.861+0000 7f2753fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:48:59.862 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:48:59.862 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:48:59.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.864+0000 7f2771c4a640 10 librados: watch_flush enter 2024-03-25T18:48:59.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.864+0000 7f2771c4a640 10 librados: watch_flush exit 2024-03-25T18:48:59.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.864+0000 7f2771c4a640 20 client.4651.objecter shutdown clearing up homeless session... 2024-03-25T18:48:59.865 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.864+0000 7f2771c4a640 10 client.4651.objecter successfully canceled tick 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.864+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 >> v1:172.21.15.136:6800/1141502808 conn(0x7f275806ddf0 legacy=0x7f27580702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.864+0000 7f2771c4a640 10 monclient: shutdown 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.864+0000 7f2771c4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.864+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 >> v1:172.21.15.136:6789/0 conn(0x7f276c0b1730 legacy=0x7f276c12fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.865+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 shutdown_connections 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.865+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 >> 172.21.15.32:0/980707052 conn(0x7f276c0617a0 msgr2=0x7f276c0b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.865+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 shutdown_connections 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.865+0000 7f2771c4a640 1 -- 172.21.15.32:0/980707052 wait complete. 2024-03-25T18:48:59.866 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:48:59.865+0000 7f2771c4a640 1 librados: shutdown 2024-03-25T18:48:59.878 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:49:06.685 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.799+0000 7f2fa6c2a640 10 monclient: get_monmap_and_config 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.799+0000 7f2fa6c2a640 10 monclient: build_initial_monmap 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.799+0000 7f2fa6c2a640 10 monclient: monmap: 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:49:06.800794+0000 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:49:06.800794+0000 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:06.800 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:06.801 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:06.801 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:49:06.801 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:49:06.801 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:06.802 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.800+0000 7f2fa6c2a640 1 Processor -- start 2024-03-25T18:49:06.802 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.800+0000 7f2fa6c2a640 1 -- start start 2024-03-25T18:49:06.802 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.800+0000 7f2fa6c2a640 10 monclient: init 2024-03-25T18:49:06.802 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:06.802 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:49:06.802 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f2fa00b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:06.802 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f2fa00b1730 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:06.802 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f2fa014be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:06.803 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa00af940 con 0x7f2fa014be90 2024-03-25T18:49:06.803 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa00b0a70 con 0x7f2fa00b1730 2024-03-25T18:49:06.803 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa00b0d50 con 0x7f2fa00b4d60 2024-03-25T18:49:06.803 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:06.803 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.801+0000 7f2fa6c2a640 10 monclient(hunting): authenticate will time out at 6652.990749s 2024-03-25T18:49:06.819 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.818+0000 7f2fa49c6640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f2fa00b4d60 0x7f2fa013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:53858/0 (socket says 172.21.15.32:53858) 2024-03-25T18:49:06.819 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.818+0000 7f2fa49c6640 1 -- 172.21.15.32:0/1882706731 learned_addr learned my addr 172.21.15.32:0/1882706731 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:06.837 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.836+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2843148088 0 0) 0x7f2fa00b0d50 con 0x7f2fa00b4d60 2024-03-25T18:49:06.837 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.836+0000 7f2f9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:49:06.837 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.836+0000 7f2f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:06.837 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.836+0000 7f2f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:06.837 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.836+0000 7f2f9f7fe640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:49:06.837 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.836+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f90003520 con 0x7f2fa00b4d60 2024-03-25T18:49:06.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.842+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 25360666 0 0) 0x7f2fa00af940 con 0x7f2fa014be90 2024-03-25T18:49:06.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.842+0000 7f2f9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:49:06.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.842+0000 7f2f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:06.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.842+0000 7f2f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:06.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.842+0000 7f2f9f7fe640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:49:06.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.842+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa00b0d50 con 0x7f2fa014be90 2024-03-25T18:49:06.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.847+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3994361859 0 0) 0x7f2fa00b0a70 con 0x7f2fa00b1730 2024-03-25T18:49:06.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.847+0000 7f2f9f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:49:06.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.847+0000 7f2f9f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:06.848 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.847+0000 7f2f9f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:06.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.847+0000 7f2f9f7fe640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:49:06.849 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.847+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa00af940 con 0x7f2fa00b1730 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3173946446 0 0) 0x7f2f90003520 con 0x7f2fa00b4d60 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 >> v1:172.21.15.136:6789/0 conn(0x7f2fa00b1730 legacy=0x7f2fa00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 >> v1:172.21.15.32:6789/0 conn(0x7f2fa014be90 legacy=0x7f2fa014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa00b1030 con 0x7f2fa00b4d60 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:49:06.850 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2f94001690 con 0x7f2fa00b4d60 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2fa6c2a640 5 monclient: authenticate success, global_id 4655 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2fa6c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2f9f7fe640 10 monclient: dump: 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:06.851 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:06.852 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.848+0000 7f2fa6c2a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:06.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.869+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f940017f0 con 0x7f2fa00b4d60 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.869+0000 7f2f9f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.869+0000 7f2f9f7fe640 1 -- 172.21.15.32:0/1882706731 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2f94001c00 con 0x7f2fa00b4d60 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.869+0000 7f2f9f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.869+0000 7f2f9f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.869+0000 7f2f9f7fe640 10 monclient: dump: 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:06.871 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:06.872 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:06.872 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:06.872 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:06.872 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.869+0000 7f2fa6c2a640 10 monclient: get_monmap_and_config success 2024-03-25T18:49:06.872 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.869+0000 7f2fa6c2a640 4 set_mon_vals no callback set 2024-03-25T18:49:06.873 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.870+0000 7f2fa6c2a640 10 monclient: shutdown 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.870+0000 7f2fa6c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.870+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/1882706731 >> v1:172.21.15.32:6790/0 conn(0x7f2fa00b4d60 legacy=0x7f2fa013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.870+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/1882706731 shutdown_connections 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.870+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/1882706731 >> 172.21.15.32:0/1882706731 conn(0x7f2fa00617a0 msgr2=0x7f2fa0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.870+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/1882706731 shutdown_connections 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.870+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/1882706731 wait complete. 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.871+0000 7f2fa6c2a640 10 monclient: build_initial_monmap 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.871+0000 7f2fa6c2a640 10 monclient: monmap: 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:49:06.874 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.872+0000 7f2fa6c2a640 1 librados: starting msgr at 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.872+0000 7f2fa6c2a640 1 librados: starting objecter 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.872+0000 7f2fa6c2a640 1 Processor -- start 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.872+0000 7f2fa6c2a640 1 -- start start 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.872+0000 7f2fa6c2a640 1 librados: setting wanted keys 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.872+0000 7f2fa6c2a640 1 librados: calling monclient init 2024-03-25T18:49:06.875 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.872+0000 7f2fa6c2a640 10 monclient: init 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.873+0000 7f2fa6c2a640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.873+0000 7f2fa6c2a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.873+0000 7f2fa6c2a640 10 monclient(hunting): picked mon.noname-a con 0x7f2fa00b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.873+0000 7f2fa6c2a640 10 monclient(hunting): picked mon.noname-c con 0x7f2fa00b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.873+0000 7f2fa6c2a640 10 monclient(hunting): picked mon.noname-b con 0x7f2fa014be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.873+0000 7f2fa6c2a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa0138010 con 0x7f2fa00b1730 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.874+0000 7f2fa6c2a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa01382f0 con 0x7f2fa014be90 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.874+0000 7f2fa6c2a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2fa0136d90 con 0x7f2fa00b4d60 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.874+0000 7f2fa6c2a640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:06.877 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.874+0000 7f2fa6c2a640 10 monclient(hunting): authenticate will time out at 6653.063634s 2024-03-25T18:49:06.883 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.882+0000 7f2fa51c7640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2fa014be90 0x7f2fa01427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:47864/0 (socket says 172.21.15.32:47864) 2024-03-25T18:49:06.883 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.882+0000 7f2fa51c7640 1 -- 172.21.15.32:0/3215938898 learned_addr learned my addr 172.21.15.32:0/3215938898 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:06.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.884+0000 7f2f9d7fa640 10 client.?.objecter ms_handle_connect 0x7f2fa014be90 2024-03-25T18:49:06.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.884+0000 7f2f9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:06.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.884+0000 7f2f9d7fa640 10 client.?.objecter ms_handle_connect 0x7f2fa00b4d60 2024-03-25T18:49:06.885 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.884+0000 7f2f9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:06.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.888+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2681234420 0 0) 0x7f2fa01382f0 con 0x7f2fa014be90 2024-03-25T18:49:06.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.888+0000 7f2f9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:49:06.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.888+0000 7f2f9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:06.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.888+0000 7f2f9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:06.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.888+0000 7f2f9d7fa640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:49:06.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.888+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2f84004eb0 con 0x7f2fa014be90 2024-03-25T18:49:06.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.895+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928505242 0 0) 0x7f2fa0136d90 con 0x7f2fa00b4d60 2024-03-25T18:49:06.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.895+0000 7f2f9d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:49:06.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.895+0000 7f2f9d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:06.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.895+0000 7f2f9d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:06.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.895+0000 7f2f9d7fa640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:49:06.896 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.895+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2fa01382f0 con 0x7f2fa00b4d60 2024-03-25T18:49:06.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.898+0000 7f2f9d7fa640 10 client.?.objecter ms_handle_connect 0x7f2fa00b1730 2024-03-25T18:49:06.899 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.898+0000 7f2f9d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:06.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.900+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1605849225 0 0) 0x7f2f84004eb0 con 0x7f2fa014be90 2024-03-25T18:49:06.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.900+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2fa0136d90 con 0x7f2fa014be90 2024-03-25T18:49:06.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.900+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2f98001c90 con 0x7f2fa014be90 2024-03-25T18:49:06.901 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.900+0000 7f2f9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.900+0000 7f2f9d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.900+0000 7f2f9d7fa640 10 monclient(hunting): dump: 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:06.902 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:06.903 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:06.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.900+0000 7f2f9d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:49:06.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.904+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 917976383 0 0) 0x7f2fa01382f0 con 0x7f2fa00b4d60 2024-03-25T18:49:06.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.904+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2f84004eb0 con 0x7f2fa00b4d60 2024-03-25T18:49:06.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.904+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2f8c0018b0 con 0x7f2fa00b4d60 2024-03-25T18:49:06.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.904+0000 7f2f9d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:06.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.904+0000 7f2f9d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:49:06.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.904+0000 7f2f9d7fa640 10 monclient(hunting): dump: 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:06.906 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1354618835 0 0) 0x7f2fa0136d90 con 0x7f2fa014be90 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 >> v1:172.21.15.32:6790/0 conn(0x7f2fa00b4d60 legacy=0x7f2fa0143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 >> v1:172.21.15.32:6789/0 conn(0x7f2fa00b1730 legacy=0x7f2fa0135f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 1 monclient: found mon.b 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa0144010 con 0x7f2fa014be90 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2f9d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:06.908 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:49:06.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 10 monclient: _renew_subs 2024-03-25T18:49:06.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:06.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fa01372b0 con 0x7f2fa014be90 2024-03-25T18:49:06.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:49:06.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 10 monclient: _renew_subs 2024-03-25T18:49:06.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:06.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fa01369c0 con 0x7f2fa014be90 2024-03-25T18:49:06.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.906+0000 7f2fa6c2a640 1 librados: init done 2024-03-25T18:49:06.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.907+0000 7f2fa6c2a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:06.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.907+0000 7f2fa6c2a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:06.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.907+0000 7f2fa6c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:06.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:06.907+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fa01375b0 con 0x7f2fa014be90 2024-03-25T18:49:07.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.117+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2f98003970 con 0x7f2fa014be90 2024-03-25T18:49:07.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.117+0000 7f2f9d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:07.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.117+0000 7f2f9effd640 4 set_mon_vals no callback set 2024-03-25T18:49:07.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.176+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2f98003cc0 con 0x7f2fa014be90 2024-03-25T18:49:07.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.176+0000 7f2f9d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:07.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.176+0000 7f2f9d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:07.177 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.176+0000 7f2f9d7fa640 10 monclient: dump: 2024-03-25T18:49:07.177 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:07.177 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:07.177 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:07.177 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:07.178 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:07.178 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:07.178 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:07.178 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:07.178 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:07.178 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:07.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.176+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f2f98003ea0 con 0x7f2fa014be90 2024-03-25T18:49:07.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.177+0000 7f2fa49c6640 10 monclient: get_auth_request con 0x7f2f8406de90 auth_method 0 2024-03-25T18:49:07.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.177+0000 7f2f9d7fa640 10 client.4657.objecter ms_handle_connect 0x7f2f8406de90 2024-03-25T18:49:07.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.181+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1010299223 0 0) 0x7f2f98054a90 con 0x7f2fa014be90 2024-03-25T18:49:07.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.181+0000 7f2f9d7fa640 10 client.4657.objecter ms_dispatch 0x7f2fa0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:49:07.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.181+0000 7f2f9d7fa640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:49:07.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.181+0000 7f2f9d7fa640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:49:07.182 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.181+0000 7f2f9d7fa640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T18:49:07.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.184+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2f980505a0 con 0x7f2fa014be90 2024-03-25T18:49:07.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.184+0000 7f2f9d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:07.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.184+0000 7f2f9d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:49:07.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.430+0000 7f2fa6c2a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:07.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.430+0000 7f2fa6c2a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:07.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.430+0000 7f2fa6c2a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:07.431 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.430+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2fa0136f20 con 0x7f2fa014be90 2024-03-25T18:49:07.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.464+0000 7f2f9d7fa640 1 -- 172.21.15.32:0/3215938898 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2566432492) 0x7f2f980524d0 con 0x7f2fa014be90 2024-03-25T18:49:07.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.464+0000 7f2f9d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:07.465 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.464+0000 7f2f9d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:49:07.466 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:49:07.466 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":10,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:39:15.535299+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:39:15.535299+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:49:07.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 10 librados: watch_flush enter 2024-03-25T18:49:07.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 10 librados: watch_flush exit 2024-03-25T18:49:07.468 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 10 client.4657.objecter successfully canceled tick 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 >> v1:172.21.15.136:6800/1141502808 conn(0x7f2f8406de90 legacy=0x7f2f84070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 10 monclient: shutdown 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 >> v1:172.21.15.136:6789/0 conn(0x7f2fa014be90 legacy=0x7f2fa01427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 shutdown_connections 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 >> 172.21.15.32:0/3215938898 conn(0x7f2fa00617a0 msgr2=0x7f2fa00b2f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 shutdown_connections 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 1 -- 172.21.15.32:0/3215938898 wait complete. 2024-03-25T18:49:07.469 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:07.467+0000 7f2fa6c2a640 1 librados: shutdown 2024-03-25T18:49:07.480 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:49:14.288 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:49:14.351 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:26.679 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.358 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:26.686 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.371 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:26.688 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.373 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:26.690 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.383 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:26.711 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.385 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:26.713 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.396 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.395+0000 7f0cfc552640 10 monclient: get_monmap_and_config 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.395+0000 7f0cfc552640 10 monclient: build_initial_monmap 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.395+0000 7f0cfc552640 10 monclient: monmap: 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:49:14.397379+0000 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:49:14.397379+0000 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:14.397 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:49:14.398 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:49:14.398 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:14.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.396+0000 7f0cfc552640 1 Processor -- start 2024-03-25T18:49:14.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.396+0000 7f0cfc552640 1 -- start start 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.396+0000 7f0cfc552640 10 monclient: init 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf40a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf41430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf4142520 con 0x7f0cf41430c0 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf40b28a0 con 0x7f0cf40a98f0 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf40b2a30 con 0x7f0cf413f710 2024-03-25T18:49:14.400 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:14.401 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.397+0000 7f0cfc552640 10 monclient(hunting): authenticate will time out at 6660.586917s 2024-03-25T18:49:14.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.419+0000 7f0cfa2ee640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0cf40a98f0 0x7f0cf40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:52232/0 (socket says 172.21.15.32:52232) 2024-03-25T18:49:14.420 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.419+0000 7f0cfa2ee640 1 -- 172.21.15.32:0/2801331748 learned_addr learned my addr 172.21.15.32:0/2801331748 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:14.428 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:26.756 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.437+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2022663645 0 0) 0x7f0cf40b28a0 con 0x7f0cf40a98f0 2024-03-25T18:49:14.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.437+0000 7f0cf92ec640 10 monclient(hunting): _negotiate 2024-03-25T18:49:14.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.437+0000 7f0cf92ec640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:14.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.437+0000 7f0cf92ec640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:14.439 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.437+0000 7f0cf92ec640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:49:14.439 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.437+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ce0003520 con 0x7f0cf40a98f0 2024-03-25T18:49:14.440 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:26.757 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.449 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:26.777 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:14.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2569728444 0 0) 0x7f0ce0003520 con 0x7f0cf40a98f0 2024-03-25T18:49:14.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 >> v1:172.21.15.32:6790/0 conn(0x7f0cf413f710 legacy=0x7f0cf41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:14.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 >> v1:172.21.15.32:6789/0 conn(0x7f0cf41430c0 legacy=0x7f0cf414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:14.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 10 monclient: _finish_hunting 0 2024-03-25T18:49:14.547 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 1 monclient: found mon.noname-c 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf40b44c0 con 0x7f0cf40a98f0 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 10 monclient: _finish_auth 0 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 10 monclient: _check_auth_tickets 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0cf0001690 con 0x7f0cf40a98f0 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cfc552640 5 monclient: authenticate success, global_id 4660 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.545+0000 7f0cfc552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.546+0000 7f0cf92ec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.546+0000 7f0cf92ec640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.546+0000 7f0cf92ec640 10 monclient: dump: 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:14.548 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:14.549 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.546+0000 7f0cfc552640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:14.560 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.559+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0cf00017f0 con 0x7f0cf40a98f0 2024-03-25T18:49:14.560 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.559+0000 7f0cf92ec640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:14.560 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.559+0000 7f0cf92ec640 1 -- 172.21.15.32:0/2801331748 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0cf0001c00 con 0x7f0cf40a98f0 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.559+0000 7f0cfc552640 10 monclient: get_monmap_and_config success 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.559+0000 7f0cfc552640 4 set_mon_vals no callback set 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.559+0000 7f0cf92ec640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.559+0000 7f0cf92ec640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.559+0000 7f0cf92ec640 10 monclient: dump: 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:14.561 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:14.562 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:14.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.560+0000 7f0cfc552640 10 monclient: shutdown 2024-03-25T18:49:14.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.560+0000 7f0cfc552640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:14.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.560+0000 7f0cfc552640 1 -- 172.21.15.32:0/2801331748 >> v1:172.21.15.136:6789/0 conn(0x7f0cf40a98f0 legacy=0x7f0cf40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:14.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.560+0000 7f0cfc552640 1 -- 172.21.15.32:0/2801331748 shutdown_connections 2024-03-25T18:49:14.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.560+0000 7f0cfc552640 1 -- 172.21.15.32:0/2801331748 >> 172.21.15.32:0/2801331748 conn(0x7f0cf40b7ea0 msgr2=0x7f0cf40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:14.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.560+0000 7f0cfc552640 1 -- 172.21.15.32:0/2801331748 shutdown_connections 2024-03-25T18:49:14.564 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.560+0000 7f0cfc552640 1 -- 172.21.15.32:0/2801331748 wait complete. 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.561+0000 7f0cfc552640 10 monclient: build_initial_monmap 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.561+0000 7f0cfc552640 10 monclient: monmap: 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.561+0000 7f0cfc552640 1 librados: starting msgr at 2024-03-25T18:49:14.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.561+0000 7f0cfc552640 1 librados: starting objecter 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 1 Processor -- start 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 1 -- start start 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 1 librados: setting wanted keys 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 1 librados: calling monclient init 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 10 monclient: init 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 10 monclient(hunting): picked mon.noname-a con 0x7f0cf40a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 10 monclient(hunting): picked mon.noname-b con 0x7f0cf413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 10 monclient(hunting): picked mon.noname-c con 0x7f0cf41430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf413be70 con 0x7f0cf40a98f0 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf4131660 con 0x7f0cf413f710 2024-03-25T18:49:14.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.562+0000 7f0cfc552640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cf41317f0 con 0x7f0cf41430c0 2024-03-25T18:49:14.567 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.563+0000 7f0cfc552640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:14.567 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.563+0000 7f0cfc552640 10 monclient(hunting): authenticate will time out at 6660.752525s 2024-03-25T18:49:14.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.576+0000 7f0cf9aed640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f0cf413f710 0x7f0cf41d76e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:52250/0 (socket says 172.21.15.32:52250) 2024-03-25T18:49:14.577 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.576+0000 7f0cf9aed640 1 -- 172.21.15.32:0/2099966390 learned_addr learned my addr 172.21.15.32:0/2099966390 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:14.578 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.577+0000 7f0ceaffd640 10 client.?.objecter ms_handle_connect 0x7f0cf413f710 2024-03-25T18:49:14.579 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.577+0000 7f0ceaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:14.586 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.585+0000 7f0ceaffd640 10 client.?.objecter ms_handle_connect 0x7f0cf40a98f0 2024-03-25T18:49:14.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.585+0000 7f0ceaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:14.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.586+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3522490360 0 0) 0x7f0cf4131660 con 0x7f0cf413f710 2024-03-25T18:49:14.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.586+0000 7f0ceaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:49:14.587 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.586+0000 7f0ceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:14.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.586+0000 7f0ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:14.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.586+0000 7f0ceaffd640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:49:14.588 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.586+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd8004eb0 con 0x7f0cf413f710 2024-03-25T18:49:14.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.601+0000 7f0ceaffd640 10 client.?.objecter ms_handle_connect 0x7f0cf41430c0 2024-03-25T18:49:14.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.601+0000 7f0ceaffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:14.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.616+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 604738424 0 0) 0x7f0cd8004eb0 con 0x7f0cf413f710 2024-03-25T18:49:14.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.616+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0cf4131660 con 0x7f0cf413f710 2024-03-25T18:49:14.617 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.616+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0ce4001a30 con 0x7f0cf413f710 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.616+0000 7f0ceaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.616+0000 7f0ceaffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.616+0000 7f0ceaffd640 10 monclient(hunting): dump: 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:14.618 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:14.619 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.616+0000 7f0ceaffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:49:14.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.623+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3026688904 0 0) 0x7f0cf41317f0 con 0x7f0cf41430c0 2024-03-25T18:49:14.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.623+0000 7f0ceaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:49:14.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.623+0000 7f0ceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:14.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.623+0000 7f0ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:14.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.623+0000 7f0ceaffd640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:49:14.625 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.623+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cd8004eb0 con 0x7f0cf41430c0 2024-03-25T18:49:14.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.633+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740565286 0 0) 0x7f0cf413be70 con 0x7f0cf40a98f0 2024-03-25T18:49:14.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.633+0000 7f0ceaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:49:14.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.633+0000 7f0ceaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:14.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.633+0000 7f0ceaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:14.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.633+0000 7f0ceaffd640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:49:14.634 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.633+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cf41317f0 con 0x7f0cf40a98f0 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2690377177 0 0) 0x7f0cf4131660 con 0x7f0cf413f710 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 >> v1:172.21.15.32:6790/0 conn(0x7f0cf41430c0 legacy=0x7f0cf41e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 >> v1:172.21.15.32:6789/0 conn(0x7f0cf40a98f0 legacy=0x7f0cf4130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 1 monclient: found mon.b 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cf4131980 con 0x7f0cf413f710 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 10 monclient: _finish_auth 0 2024-03-25T18:49:14.645 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0ceaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 5 monclient: authenticate success, global_id 4663 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 10 monclient: _renew_subs 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cf4135c60 con 0x7f0cf413f710 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 10 client.4663.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 10 monclient: _renew_subs 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cf40b7770 con 0x7f0cf413f710 2024-03-25T18:49:14.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.643+0000 7f0cfc552640 1 librados: init done 2024-03-25T18:49:14.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.644+0000 7f0cfc552640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:14.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.644+0000 7f0cfc552640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:14.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.644+0000 7f0cfc552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:14.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.644+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cf41371d0 con 0x7f0cf413f710 2024-03-25T18:49:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.947+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ce4001cb0 con 0x7f0cf413f710 2024-03-25T18:49:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.947+0000 7f0ceaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.947+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f0ce4003a60 con 0x7f0cf413f710 2024-03-25T18:49:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.947+0000 7f0ceaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.947+0000 7f0ceaffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.947+0000 7f0cf8aeb640 4 set_mon_vals no callback set 2024-03-25T18:49:14.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.947+0000 7f0ceaffd640 10 monclient: dump: 2024-03-25T18:49:14.949 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:14.950 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:14.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.948+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f0ce4003c80 con 0x7f0cf413f710 2024-03-25T18:49:14.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.949+0000 7f0cfa2ee640 10 monclient: get_auth_request con 0x7f0cd806dea0 auth_method 0 2024-03-25T18:49:14.951 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.949+0000 7f0ceaffd640 10 client.4663.objecter ms_handle_connect 0x7f0cd806dea0 2024-03-25T18:49:14.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.954+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7f0ce40548d0 con 0x7f0cf413f710 2024-03-25T18:49:14.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.954+0000 7f0ceaffd640 10 client.4663.objecter ms_dispatch 0x7f0cf4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:49:14.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.954+0000 7f0ceaffd640 3 client.4663.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:49:14.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.954+0000 7f0ceaffd640 3 client.4663.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:49:14.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.954+0000 7f0ceaffd640 20 client.4663.objecter dump_active .. 0 homeless 2024-03-25T18:49:14.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.954+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0ce40503e0 con 0x7f0cf413f710 2024-03-25T18:49:14.955 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.954+0000 7f0ceaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:14.956 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:14.954+0000 7f0ceaffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:49:15.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.207+0000 7f0cfc552640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:15.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.207+0000 7f0cfc552640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:15.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.207+0000 7f0cfc552640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:15.208 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.207+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0cf40627b0 con 0x7f0cf413f710 2024-03-25T18:49:15.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.287+0000 7f0ceaffd640 1 -- 172.21.15.32:0/2099966390 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7f0ce4052310 con 0x7f0cf413f710 2024-03-25T18:49:15.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.287+0000 7f0ceaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:15.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.287+0000 7f0ceaffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:49:15.289 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:49:15.289 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:49:15.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.289+0000 7f0cfc552640 10 librados: watch_flush enter 2024-03-25T18:49:15.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.289+0000 7f0cfc552640 10 librados: watch_flush exit 2024-03-25T18:49:15.291 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.289+0000 7f0cfc552640 20 client.4663.objecter shutdown clearing up homeless session... 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.289+0000 7f0cfc552640 10 client.4663.objecter successfully canceled tick 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.289+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 >> v1:172.21.15.136:6800/1141502808 conn(0x7f0cd806dea0 legacy=0x7f0cd8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.289+0000 7f0cfc552640 10 monclient: shutdown 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.289+0000 7f0cfc552640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.289+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 >> v1:172.21.15.136:6789/0 conn(0x7f0cf413f710 legacy=0x7f0cf41d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.290+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 shutdown_connections 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.290+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 >> 172.21.15.32:0/2099966390 conn(0x7f0cf40b7ea0 msgr2=0x7f0cf4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.290+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 shutdown_connections 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.290+0000 7f0cfc552640 1 -- 172.21.15.32:0/2099966390 wait complete. 2024-03-25T18:49:15.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:15.290+0000 7f0cfc552640 1 librados: shutdown 2024-03-25T18:49:15.302 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:49:22.108 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:49:22.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.224+0000 7fee3744e640 10 monclient: get_monmap_and_config 2024-03-25T18:49:22.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.224+0000 7fee3744e640 10 monclient: build_initial_monmap 2024-03-25T18:49:22.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.224+0000 7fee3744e640 10 monclient: monmap: 2024-03-25T18:49:22.225 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:49:22.226121+0000 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:49:22.226121+0000 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:49:22.226 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:22.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.225+0000 7fee3744e640 1 Processor -- start 2024-03-25T18:49:22.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.225+0000 7fee3744e640 1 -- start start 2024-03-25T18:49:22.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.225+0000 7fee3744e640 10 monclient: init 2024-03-25T18:49:22.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:22.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:49:22.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 10 monclient(hunting): picked mon.noname-a con 0x7fee300b4d60 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:22.227 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 10 monclient(hunting): picked mon.noname-b con 0x7fee300b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:22.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 10 monclient(hunting): picked mon.noname-c con 0x7fee3014be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:22.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee300af940 con 0x7fee300b4d60 2024-03-25T18:49:22.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee300b0a70 con 0x7fee3014be90 2024-03-25T18:49:22.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee300b0d50 con 0x7fee300b1730 2024-03-25T18:49:22.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:22.228 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.226+0000 7fee3744e640 10 monclient(hunting): authenticate will time out at 6668.415766s 2024-03-25T18:49:22.235 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.234+0000 7fee359eb640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fee3014be90 0x7fee3014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:54962/0 (socket says 172.21.15.32:54962) 2024-03-25T18:49:22.235 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.234+0000 7fee359eb640 1 -- 172.21.15.32:0/3491614335 learned_addr learned my addr 172.21.15.32:0/3491614335 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:22.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.250+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1928300873 0 0) 0x7fee300b0a70 con 0x7fee3014be90 2024-03-25T18:49:22.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.250+0000 7fee27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:49:22.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.250+0000 7fee27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:22.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.250+0000 7fee27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:22.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.250+0000 7fee27fff640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:49:22.251 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.250+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee1c003520 con 0x7fee3014be90 2024-03-25T18:49:22.253 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.252+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 902252700 0 0) 0x7fee300af940 con 0x7fee300b4d60 2024-03-25T18:49:22.253 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.252+0000 7fee27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:49:22.253 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.252+0000 7fee27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:22.253 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.252+0000 7fee27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:22.253 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.252+0000 7fee27fff640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:49:22.254 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.252+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee300b0a70 con 0x7fee300b4d60 2024-03-25T18:49:22.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.260+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 470974941 0 0) 0x7fee300b0d50 con 0x7fee300b1730 2024-03-25T18:49:22.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.260+0000 7fee27fff640 10 monclient(hunting): _negotiate 2024-03-25T18:49:22.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.260+0000 7fee27fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:22.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.260+0000 7fee27fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:22.262 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.260+0000 7fee27fff640 10 monclient(hunting): my global_id is 4667 2024-03-25T18:49:22.262 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.260+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee300af940 con 0x7fee300b1730 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.265+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1674484019 0 0) 0x7fee1c003520 con 0x7fee3014be90 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.265+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 >> v1:172.21.15.32:6790/0 conn(0x7fee300b1730 legacy=0x7fee300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.265+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 >> v1:172.21.15.32:6789/0 conn(0x7fee300b4d60 legacy=0x7fee3013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 10 monclient: _finish_hunting 0 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 1 monclient: found mon.noname-c 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee300b1030 con 0x7fee3014be90 2024-03-25T18:49:22.267 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 10 monclient: _finish_auth 0 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 10 monclient: _check_auth_tickets 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fee28001a10 con 0x7fee3014be90 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee27fff640 10 monclient: dump: 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:22.268 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:22.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee3744e640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:49:22.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.266+0000 7fee3744e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:22.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.285+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee28001b70 con 0x7fee3014be90 2024-03-25T18:49:22.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.285+0000 7fee27fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:22.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.285+0000 7fee27fff640 1 -- 172.21.15.32:0/3491614335 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fee28001f80 con 0x7fee3014be90 2024-03-25T18:49:22.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.285+0000 7fee3744e640 10 monclient: get_monmap_and_config success 2024-03-25T18:49:22.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.285+0000 7fee3744e640 4 set_mon_vals no callback set 2024-03-25T18:49:22.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.285+0000 7fee27fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:22.286 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.285+0000 7fee27fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.285+0000 7fee27fff640 10 monclient: dump: 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:22.287 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:22.288 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.286+0000 7fee3744e640 10 monclient: shutdown 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.286+0000 7fee3744e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.286+0000 7fee3744e640 1 -- 172.21.15.32:0/3491614335 >> v1:172.21.15.136:6789/0 conn(0x7fee3014be90 legacy=0x7fee3014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.286+0000 7fee3744e640 1 -- 172.21.15.32:0/3491614335 shutdown_connections 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.286+0000 7fee3744e640 1 -- 172.21.15.32:0/3491614335 >> 172.21.15.32:0/3491614335 conn(0x7fee300617a0 msgr2=0x7fee30061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.286+0000 7fee3744e640 1 -- 172.21.15.32:0/3491614335 shutdown_connections 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.286+0000 7fee3744e640 1 -- 172.21.15.32:0/3491614335 wait complete. 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 10 monclient: build_initial_monmap 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 10 monclient: monmap: 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:22.289 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 1 librados: starting msgr at 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 1 librados: starting objecter 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 1 Processor -- start 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 1 -- start start 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 1 librados: setting wanted keys 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 1 librados: calling monclient init 2024-03-25T18:49:22.290 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.287+0000 7fee3744e640 10 monclient: init 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 10 monclient(hunting): picked mon.noname-c con 0x7fee300b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 10 monclient(hunting): picked mon.noname-b con 0x7fee300b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 10 monclient(hunting): picked mon.noname-a con 0x7fee3014be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee30138640 con 0x7fee3014be90 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee30139800 con 0x7fee300b4d60 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee30132620 con 0x7fee300b1730 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.288+0000 7fee3744e640 10 monclient(hunting): authenticate will time out at 6668.478148s 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.289+0000 7fee349e9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fee300b4d60 0x7fee3013b040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:55016/0 (socket says 172.21.15.32:55016) 2024-03-25T18:49:22.292 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.289+0000 7fee349e9640 1 -- 172.21.15.32:0/48143278 learned_addr learned my addr 172.21.15.32:0/48143278 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:22.293 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.289+0000 7fee25ffb640 10 client.?.objecter ms_handle_connect 0x7fee300b4d60 2024-03-25T18:49:22.293 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.289+0000 7fee25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:22.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.303+0000 7fee25ffb640 10 client.?.objecter ms_handle_connect 0x7fee300b1730 2024-03-25T18:49:22.304 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.303+0000 7fee25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:22.364 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.363+0000 7fee25ffb640 10 client.?.objecter ms_handle_connect 0x7fee3014be90 2024-03-25T18:49:22.364 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.363+0000 7fee25ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:22.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.385+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321099734 0 0) 0x7fee30138640 con 0x7fee3014be90 2024-03-25T18:49:22.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.385+0000 7fee25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:49:22.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.385+0000 7fee25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:22.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.385+0000 7fee25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:22.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.385+0000 7fee25ffb640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:49:22.386 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.385+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee14004eb0 con 0x7fee3014be90 2024-03-25T18:49:22.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.411+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2411164327 0 0) 0x7fee14004eb0 con 0x7fee3014be90 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.411+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee30138640 con 0x7fee3014be90 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.411+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fee28001e80 con 0x7fee3014be90 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.411+0000 7fee25ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.411+0000 7fee25ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.411+0000 7fee25ffb640 10 monclient(hunting): dump: 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:22.413 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:22.414 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.411+0000 7fee25ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:49:22.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.423+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4130310459 0 0) 0x7fee30139800 con 0x7fee300b4d60 2024-03-25T18:49:22.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.423+0000 7fee25ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:49:22.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.423+0000 7fee25ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:22.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.423+0000 7fee25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:22.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.423+0000 7fee25ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:49:22.424 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.423+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee14004eb0 con 0x7fee300b4d60 2024-03-25T18:49:22.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1913291946 0 0) 0x7fee30138640 con 0x7fee3014be90 2024-03-25T18:49:22.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 >> v1:172.21.15.32:6790/0 conn(0x7fee300b1730 legacy=0x7fee30131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:22.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 >> v1:172.21.15.136:6789/0 conn(0x7fee300b4d60 legacy=0x7fee3013b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:22.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:49:22.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 1 monclient: found mon.a 2024-03-25T18:49:22.433 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee301327b0 con 0x7fee3014be90 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 10 monclient: _finish_auth 0 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 10 monclient: _renew_subs 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee30138dd0 con 0x7fee3014be90 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 10 client.4695.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 10 monclient: _renew_subs 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee30138270 con 0x7fee3014be90 2024-03-25T18:49:22.434 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.432+0000 7fee3744e640 1 librados: init done 2024-03-25T18:49:22.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.433+0000 7fee3744e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:22.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.433+0000 7fee3744e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:22.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.433+0000 7fee3744e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:22.435 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.433+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee300b2270 con 0x7fee3014be90 2024-03-25T18:49:22.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.702+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee28002060 con 0x7fee3014be90 2024-03-25T18:49:22.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.702+0000 7fee25ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:22.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.702+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fee28004fb0 con 0x7fee3014be90 2024-03-25T18:49:22.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.702+0000 7fee25ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:22.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.702+0000 7fee25ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:49:22.703 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.702+0000 7fee25ffb640 10 monclient: dump: 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:22.704 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.702+0000 7fee277fe640 4 set_mon_vals no callback set 2024-03-25T18:49:22.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.702+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fee2801af80 con 0x7fee3014be90 2024-03-25T18:49:22.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.703+0000 7fee351ea640 10 monclient: get_auth_request con 0x7fee1406dd40 auth_method 0 2024-03-25T18:49:22.705 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.703+0000 7fee25ffb640 10 client.4695.objecter ms_handle_connect 0x7fee1406dd40 2024-03-25T18:49:22.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.709+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7fee280549e0 con 0x7fee3014be90 2024-03-25T18:49:22.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.709+0000 7fee25ffb640 10 client.4695.objecter ms_dispatch 0x7fee30000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:49:22.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.709+0000 7fee25ffb640 3 client.4695.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:49:22.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.709+0000 7fee25ffb640 3 client.4695.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:49:22.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.709+0000 7fee25ffb640 20 client.4695.objecter dump_active .. 0 homeless 2024-03-25T18:49:22.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.709+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fee280504f0 con 0x7fee3014be90 2024-03-25T18:49:22.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.709+0000 7fee25ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:22.710 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.709+0000 7fee25ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:49:22.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.933+0000 7fee3744e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:22.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.933+0000 7fee3744e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:22.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.933+0000 7fee3744e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:22.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.933+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fee301387d0 con 0x7fee3014be90 2024-03-25T18:49:22.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.962+0000 7fee25ffb640 1 -- 172.21.15.32:0/48143278 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7fee28052420 con 0x7fee3014be90 2024-03-25T18:49:22.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.962+0000 7fee25ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:22.963 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.962+0000 7fee25ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:49:22.963 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:49:22.963 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:49:22.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 10 librados: watch_flush enter 2024-03-25T18:49:22.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 10 librados: watch_flush exit 2024-03-25T18:49:22.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 20 client.4695.objecter shutdown clearing up homeless session... 2024-03-25T18:49:22.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 10 client.4695.objecter successfully canceled tick 2024-03-25T18:49:22.966 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 >> v1:172.21.15.136:6800/1141502808 conn(0x7fee1406dd40 legacy=0x7fee14070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:22.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 10 monclient: shutdown 2024-03-25T18:49:22.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:22.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 >> v1:172.21.15.32:6789/0 conn(0x7fee3014be90 legacy=0x7fee3013b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:22.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 shutdown_connections 2024-03-25T18:49:22.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 >> 172.21.15.32:0/48143278 conn(0x7fee300617a0 msgr2=0x7fee3013c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:22.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 shutdown_connections 2024-03-25T18:49:22.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.964+0000 7fee3744e640 1 -- 172.21.15.32:0/48143278 wait complete. 2024-03-25T18:49:22.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:22.965+0000 7fee3744e640 1 librados: shutdown 2024-03-25T18:49:22.977 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:49:29.785 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:49:29.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.901+0000 7fd7ab633640 10 monclient: get_monmap_and_config 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.901+0000 7fd7ab633640 10 monclient: build_initial_monmap 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.901+0000 7fd7ab633640 10 monclient: monmap: 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:49:29.903377+0000 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:49:29.903377+0000 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:49:29.903 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.902+0000 7fd7ab633640 1 Processor -- start 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.902+0000 7fd7ab633640 1 -- start start 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.902+0000 7fd7ab633640 10 monclient: init 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a40a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a41430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a4142520 con 0x7fd7a40a98f0 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40b28a0 con 0x7fd7a413f710 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a40b2a30 con 0x7fd7a41430c0 2024-03-25T18:49:29.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:29.906 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.903+0000 7fd7ab633640 10 monclient(hunting): authenticate will time out at 6676.093123s 2024-03-25T18:49:29.915 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.914+0000 7fd7a93cf640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd7a40a98f0 0x7fd7a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37362/0 (socket says 172.21.15.32:37362) 2024-03-25T18:49:29.915 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.914+0000 7fd7a93cf640 1 -- 172.21.15.32:0/2685692046 learned_addr learned my addr 172.21.15.32:0/2685692046 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:29.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.931+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4075496950 0 0) 0x7fd7a4142520 con 0x7fd7a40a98f0 2024-03-25T18:49:29.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.931+0000 7fd79bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:49:29.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.931+0000 7fd79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:29.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.931+0000 7fd79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:29.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.931+0000 7fd79bfff640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:49:29.932 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.931+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd790003520 con 0x7fd7a40a98f0 2024-03-25T18:49:29.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.933+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3283600050 0 0) 0x7fd7a40b2a30 con 0x7fd7a41430c0 2024-03-25T18:49:29.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.933+0000 7fd79bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:49:29.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.933+0000 7fd79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:29.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.933+0000 7fd79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:29.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.933+0000 7fd79bfff640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:49:29.934 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.933+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a4142520 con 0x7fd7a41430c0 2024-03-25T18:49:29.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.935+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2425597662 0 0) 0x7fd7a40b28a0 con 0x7fd7a413f710 2024-03-25T18:49:29.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.935+0000 7fd79bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:49:29.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.935+0000 7fd79bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:29.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.935+0000 7fd79bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:29.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.935+0000 7fd79bfff640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:49:29.936 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.935+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a40b2a30 con 0x7fd7a413f710 2024-03-25T18:49:29.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2919116008 0 0) 0x7fd7a40b2a30 con 0x7fd7a413f710 2024-03-25T18:49:29.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 >> v1:172.21.15.32:6790/0 conn(0x7fd7a41430c0 legacy=0x7fd7a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:29.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 >> v1:172.21.15.32:6789/0 conn(0x7fd7a40a98f0 legacy=0x7fd7a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:29.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:49:29.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 1 monclient: found mon.noname-c 2024-03-25T18:49:29.947 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a40b44c0 con 0x7fd7a413f710 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 10 monclient: _finish_auth 0 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd7940015b0 con 0x7fd7a413f710 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd7ab633640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd7ab633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd79bfff640 10 monclient: dump: 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:29.948 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:29.949 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:29.949 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:29.949 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:29.949 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:29.949 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:29.949 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:29.949 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:29.949 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.946+0000 7fd7ab633640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:29.967 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.966+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd794001710 con 0x7fd7a413f710 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.966+0000 7fd79bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.966+0000 7fd79bfff640 1 -- 172.21.15.32:0/2685692046 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd794001b60 con 0x7fd7a413f710 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.966+0000 7fd79bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.966+0000 7fd79bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.966+0000 7fd79bfff640 10 monclient: dump: 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:29.968 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:29.969 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:29.969 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.966+0000 7fd7ab633640 10 monclient: get_monmap_and_config success 2024-03-25T18:49:29.969 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.966+0000 7fd7ab633640 4 set_mon_vals no callback set 2024-03-25T18:49:29.971 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.967+0000 7fd7ab633640 10 monclient: shutdown 2024-03-25T18:49:29.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.967+0000 7fd7ab633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:29.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.967+0000 7fd7ab633640 1 -- 172.21.15.32:0/2685692046 >> v1:172.21.15.136:6789/0 conn(0x7fd7a413f710 legacy=0x7fd7a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:29.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.967+0000 7fd7ab633640 1 -- 172.21.15.32:0/2685692046 shutdown_connections 2024-03-25T18:49:29.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.967+0000 7fd7ab633640 1 -- 172.21.15.32:0/2685692046 >> 172.21.15.32:0/2685692046 conn(0x7fd7a40b7ea0 msgr2=0x7fd7a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:29.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.967+0000 7fd7ab633640 1 -- 172.21.15.32:0/2685692046 shutdown_connections 2024-03-25T18:49:29.972 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.967+0000 7fd7ab633640 1 -- 172.21.15.32:0/2685692046 wait complete. 2024-03-25T18:49:29.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.968+0000 7fd7ab633640 10 monclient: build_initial_monmap 2024-03-25T18:49:29.973 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.968+0000 7fd7ab633640 10 monclient: monmap: 2024-03-25T18:49:29.973 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:29.973 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:49:29.973 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:49:29.973 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:49:29.973 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:29.973 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:49:29.974 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:29.974 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:49:29.974 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:49:29.974 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:29.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 librados: starting msgr at 2024-03-25T18:49:29.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 librados: starting objecter 2024-03-25T18:49:29.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 Processor -- start 2024-03-25T18:49:29.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 -- start start 2024-03-25T18:49:29.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 librados: setting wanted keys 2024-03-25T18:49:29.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 librados: calling monclient init 2024-03-25T18:49:29.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 10 monclient: init 2024-03-25T18:49:29.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:29.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:49:29.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 10 monclient(hunting): picked mon.noname-c con 0x7fd7a40a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:29.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 10 monclient(hunting): picked mon.noname-b con 0x7fd7a413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:29.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 10 monclient(hunting): picked mon.noname-a con 0x7fd7a41430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:29.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a41db910 con 0x7fd7a41430c0 2024-03-25T18:49:29.976 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a41db410 con 0x7fd7a413f710 2024-03-25T18:49:29.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7a41db5a0 con 0x7fd7a40a98f0 2024-03-25T18:49:29.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:29.977 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.969+0000 7fd7ab633640 10 monclient(hunting): authenticate will time out at 6676.159319s 2024-03-25T18:49:29.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.985+0000 7fd7a9bd0640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fd7a41430c0 0x7fd7a4137a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:37402/0 (socket says 172.21.15.32:37402) 2024-03-25T18:49:29.986 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.985+0000 7fd7a9bd0640 1 -- 172.21.15.32:0/431292169 learned_addr learned my addr 172.21.15.32:0/431292169 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:29.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.987+0000 7fd799ffb640 10 client.?.objecter ms_handle_connect 0x7fd7a41430c0 2024-03-25T18:49:29.988 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.987+0000 7fd799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:29.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.989+0000 7fd799ffb640 10 client.?.objecter ms_handle_connect 0x7fd7a413f710 2024-03-25T18:49:29.990 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:29.989+0000 7fd799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:30.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.001+0000 7fd799ffb640 10 client.?.objecter ms_handle_connect 0x7fd7a40a98f0 2024-03-25T18:49:30.002 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.001+0000 7fd799ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:30.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.008+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2687570158 0 0) 0x7fd7a41db910 con 0x7fd7a41430c0 2024-03-25T18:49:30.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.008+0000 7fd799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:49:30.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.008+0000 7fd799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:30.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.008+0000 7fd799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:30.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.008+0000 7fd799ffb640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:49:30.009 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.008+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd788004eb0 con 0x7fd7a41430c0 2024-03-25T18:49:30.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.009+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3265465185 0 0) 0x7fd7a41db410 con 0x7fd7a413f710 2024-03-25T18:49:30.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.009+0000 7fd799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:49:30.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.009+0000 7fd799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:30.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.009+0000 7fd799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:30.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.009+0000 7fd799ffb640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:49:30.010 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.009+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a41db910 con 0x7fd7a413f710 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 769431153 0 0) 0x7fd7a41db910 con 0x7fd7a413f710 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a41db410 con 0x7fd7a413f710 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd794001a20 con 0x7fd7a413f710 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 10 monclient(hunting): dump: 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:30.026 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:30.027 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:30.027 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:30.027 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:30.027 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:30.027 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:30.027 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:30.027 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259279160 0 0) 0x7fd7a41db5a0 con 0x7fd7a40a98f0 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.025+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd7a41db910 con 0x7fd7a40a98f0 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.027+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 593061212 0 0) 0x7fd788004eb0 con 0x7fd7a41430c0 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.027+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd7a41db5a0 con 0x7fd7a41430c0 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.027+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd79c001c90 con 0x7fd7a41430c0 2024-03-25T18:49:30.028 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.027+0000 7fd799ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.027+0000 7fd799ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.027+0000 7fd799ffb640 10 monclient(hunting): dump: 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:30.029 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.038+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2076225074 0 0) 0x7fd7a41db910 con 0x7fd7a40a98f0 2024-03-25T18:49:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.038+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd788004eb0 con 0x7fd7a40a98f0 2024-03-25T18:49:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.038+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd7a00018d0 con 0x7fd7a40a98f0 2024-03-25T18:49:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.038+0000 7fd799ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.038+0000 7fd799ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:49:30.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.038+0000 7fd799ffb640 10 monclient(hunting): dump: 2024-03-25T18:49:30.039 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:30.040 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:30.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3683222545 0 0) 0x7fd7a41db410 con 0x7fd7a413f710 2024-03-25T18:49:30.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 >> v1:172.21.15.32:6790/0 conn(0x7fd7a40a98f0 legacy=0x7fd7a41dad00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:30.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 >> v1:172.21.15.32:6789/0 conn(0x7fd7a41430c0 legacy=0x7fd7a4137a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:30.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:49:30.043 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 1 monclient: found mon.b 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7a4134690 con 0x7fd7a413f710 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 10 monclient: _finish_auth 0 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd799ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd7ab633640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd7ab633640 10 monclient: _renew_subs 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd7ab633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.041+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7a4134960 con 0x7fd7a413f710 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.042+0000 7fd7ab633640 10 client.4675.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.042+0000 7fd7ab633640 10 monclient: _renew_subs 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.042+0000 7fd7ab633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.042+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7a4138b60 con 0x7fd7a413f710 2024-03-25T18:49:30.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.042+0000 7fd7ab633640 1 librados: init done 2024-03-25T18:49:30.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.043+0000 7fd7ab633640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:30.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.043+0000 7fd7ab633640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:30.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.043+0000 7fd7ab633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:30.045 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.043+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7a4074cc0 con 0x7fd7a413f710 2024-03-25T18:49:30.325 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.324+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd794001be0 con 0x7fd7a413f710 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.324+0000 7fd799ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.324+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fd794003a40 con 0x7fd7a413f710 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.324+0000 7fd799ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.324+0000 7fd799ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.324+0000 7fd799ffb640 10 monclient: dump: 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:30.326 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:30.327 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:30.327 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:30.327 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:30.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.324+0000 7fd79b7fe640 4 set_mon_vals no callback set 2024-03-25T18:49:30.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.325+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fd794019a10 con 0x7fd7a413f710 2024-03-25T18:49:30.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.326+0000 7fd7a93cf640 10 monclient: get_auth_request con 0x7fd78806e160 auth_method 0 2024-03-25T18:49:30.327 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.326+0000 7fd799ffb640 10 client.4675.objecter ms_handle_connect 0x7fd78806e160 2024-03-25T18:49:30.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.329+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7fd7940535b0 con 0x7fd7a413f710 2024-03-25T18:49:30.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.329+0000 7fd799ffb640 10 client.4675.objecter ms_dispatch 0x7fd7a4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:49:30.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.329+0000 7fd799ffb640 3 client.4675.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:49:30.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.329+0000 7fd799ffb640 3 client.4675.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:49:30.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.329+0000 7fd799ffb640 20 client.4675.objecter dump_active .. 0 homeless 2024-03-25T18:49:30.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.329+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd79404eef0 con 0x7fd7a413f710 2024-03-25T18:49:30.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.329+0000 7fd799ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:30.331 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.329+0000 7fd799ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:49:30.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.564+0000 7fd7ab633640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:30.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.564+0000 7fd7ab633640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:30.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.564+0000 7fd7ab633640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:30.566 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.564+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7a413e050 con 0x7fd7a413f710 2024-03-25T18:49:30.656 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.655+0000 7fd799ffb640 1 -- 172.21.15.32:0/431292169 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7fd794050ff0 con 0x7fd7a413f710 2024-03-25T18:49:30.656 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.655+0000 7fd799ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:30.656 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.655+0000 7fd799ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:49:30.657 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:49:30.657 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:49:30.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.657+0000 7fd7ab633640 10 librados: watch_flush enter 2024-03-25T18:49:30.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.657+0000 7fd7ab633640 10 librados: watch_flush exit 2024-03-25T18:49:30.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.657+0000 7fd7ab633640 20 client.4675.objecter shutdown clearing up homeless session... 2024-03-25T18:49:30.659 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.657+0000 7fd7ab633640 10 client.4675.objecter successfully canceled tick 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 >> v1:172.21.15.136:6800/1141502808 conn(0x7fd78806e160 legacy=0x7fd788070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 10 monclient: shutdown 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 >> v1:172.21.15.136:6789/0 conn(0x7fd7a413f710 legacy=0x7fd7a4133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 shutdown_connections 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 >> 172.21.15.32:0/431292169 conn(0x7fd7a40b7ea0 msgr2=0x7fd7a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 shutdown_connections 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 1 -- 172.21.15.32:0/431292169 wait complete. 2024-03-25T18:49:30.660 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:30.658+0000 7fd7ab633640 1 librados: shutdown 2024-03-25T18:49:30.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:49:34.212 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:46.539 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:34.264 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:46.592 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:34.308 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:10:46.636 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:34.315 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:46.632 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:34.317 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:46.645 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:34.321 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:46.638 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:34.330 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:46.658 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:34.343 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:10:46.660 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:34.360 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:10:46.688 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:37.477 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:49:37.594 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.594+0000 7fc51bd5d640 10 monclient: get_monmap_and_config 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.594+0000 7fc51bd5d640 10 monclient: build_initial_monmap 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.594+0000 7fc51bd5d640 10 monclient: monmap: 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:49:37.595329+0000 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:49:37.595329+0000 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:49:37.595 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:49:37.596 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:37.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.595+0000 7fc51bd5d640 1 Processor -- start 2024-03-25T18:49:37.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.595+0000 7fc51bd5d640 1 -- start start 2024-03-25T18:49:37.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.595+0000 7fc51bd5d640 10 monclient: init 2024-03-25T18:49:37.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:37.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:49:37.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5140a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:37.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc51413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:37.597 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc5141430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:37.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc514142520 con 0x7fc5141430c0 2024-03-25T18:49:37.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5140b28a0 con 0x7fc5140a98f0 2024-03-25T18:49:37.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5140b2a30 con 0x7fc51413f710 2024-03-25T18:49:37.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:37.598 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.596+0000 7fc51bd5d640 10 monclient(hunting): authenticate will time out at 6683.785093s 2024-03-25T18:49:37.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.606+0000 7fc519af9640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fc5140a98f0 0x7fc5140a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:60184/0 (socket says 172.21.15.32:60184) 2024-03-25T18:49:37.607 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.606+0000 7fc519af9640 1 -- 172.21.15.32:0/4174040948 learned_addr learned my addr 172.21.15.32:0/4174040948 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:37.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.614+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1268612230 0 0) 0x7fc5140b28a0 con 0x7fc5140a98f0 2024-03-25T18:49:37.614 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.614+0000 7fc518af7640 10 monclient(hunting): _negotiate 2024-03-25T18:49:37.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.614+0000 7fc518af7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:37.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.614+0000 7fc518af7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:37.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.614+0000 7fc518af7640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:49:37.615 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.614+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc500003520 con 0x7fc5140a98f0 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1726134297 0 0) 0x7fc500003520 con 0x7fc5140a98f0 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 >> v1:172.21.15.32:6790/0 conn(0x7fc51413f710 legacy=0x7fc5141429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 >> v1:172.21.15.32:6789/0 conn(0x7fc5141430c0 legacy=0x7fc51414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 10 monclient: _finish_hunting 0 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 1 monclient: found mon.noname-c 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5140b44c0 con 0x7fc5140a98f0 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 10 monclient: _finish_auth 0 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 10 monclient: _check_auth_tickets 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:37.627 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc510001690 con 0x7fc5140a98f0 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc518af7640 10 monclient: dump: 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:37.628 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:37.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc51bd5d640 5 monclient: authenticate success, global_id 4678 2024-03-25T18:49:37.629 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.626+0000 7fc51bd5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.634+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5100017f0 con 0x7fc5140a98f0 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.634+0000 7fc518af7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.634+0000 7fc518af7640 1 -- 172.21.15.32:0/4174040948 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc510001c40 con 0x7fc5140a98f0 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.634+0000 7fc51bd5d640 10 monclient: get_monmap_and_config success 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.634+0000 7fc51bd5d640 4 set_mon_vals no callback set 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.634+0000 7fc518af7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.634+0000 7fc518af7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.634+0000 7fc518af7640 10 monclient: dump: 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:37.635 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:37.636 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:37.636 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:37.636 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:37.636 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:37.636 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:37.636 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:37.636 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:37.638 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.635+0000 7fc51bd5d640 10 monclient: shutdown 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.635+0000 7fc51bd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.635+0000 7fc51bd5d640 1 -- 172.21.15.32:0/4174040948 >> v1:172.21.15.136:6789/0 conn(0x7fc5140a98f0 legacy=0x7fc5140a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.635+0000 7fc51bd5d640 1 -- 172.21.15.32:0/4174040948 shutdown_connections 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.635+0000 7fc51bd5d640 1 -- 172.21.15.32:0/4174040948 >> 172.21.15.32:0/4174040948 conn(0x7fc5140b7ea0 msgr2=0x7fc5140b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.636+0000 7fc51bd5d640 1 -- 172.21.15.32:0/4174040948 shutdown_connections 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.636+0000 7fc51bd5d640 1 -- 172.21.15.32:0/4174040948 wait complete. 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.636+0000 7fc51bd5d640 10 monclient: build_initial_monmap 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.636+0000 7fc51bd5d640 10 monclient: monmap: 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:49:37.639 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 1 librados: starting msgr at 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 1 librados: starting objecter 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 1 Processor -- start 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 1 -- start start 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 1 librados: setting wanted keys 2024-03-25T18:49:37.640 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 1 librados: calling monclient init 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 10 monclient: init 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 10 monclient(hunting): picked mon.noname-b con 0x7fc5140a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.637+0000 7fc51bd5d640 10 monclient(hunting): picked mon.noname-a con 0x7fc51413f710 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.638+0000 7fc51bd5d640 10 monclient(hunting): picked mon.noname-c con 0x7fc5141430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.638+0000 7fc51bd5d640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc514138b20 con 0x7fc51413f710 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.638+0000 7fc51bd5d640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc514137a80 con 0x7fc5140a98f0 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.638+0000 7fc51bd5d640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc514139880 con 0x7fc5141430c0 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.638+0000 7fc51bd5d640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:37.641 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.638+0000 7fc51bd5d640 10 monclient(hunting): authenticate will time out at 6683.826630s 2024-03-25T18:49:37.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.646+0000 7fc51a2fa640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fc5141430c0 0x7fc5141e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:45148/0 (socket says 172.21.15.32:45148) 2024-03-25T18:49:37.646 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.646+0000 7fc51a2fa640 1 -- 172.21.15.32:0/170863849 learned_addr learned my addr 172.21.15.32:0/170863849 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:37.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.647+0000 7fc50a7fc640 10 client.?.objecter ms_handle_connect 0x7fc5141430c0 2024-03-25T18:49:37.647 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.647+0000 7fc50a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:37.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.652+0000 7fc50a7fc640 10 client.?.objecter ms_handle_connect 0x7fc5140a98f0 2024-03-25T18:49:37.652 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.652+0000 7fc50a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:37.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.654+0000 7fc50a7fc640 10 client.?.objecter ms_handle_connect 0x7fc51413f710 2024-03-25T18:49:37.654 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.654+0000 7fc50a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:37.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.657+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 272911073 0 0) 0x7fc514137a80 con 0x7fc5140a98f0 2024-03-25T18:49:37.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.657+0000 7fc50a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:49:37.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.657+0000 7fc50a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:37.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.657+0000 7fc50a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:37.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.657+0000 7fc50a7fc640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:49:37.658 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.658+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc4f8004eb0 con 0x7fc5140a98f0 2024-03-25T18:49:37.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.661+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394271825 0 0) 0x7fc514139880 con 0x7fc5141430c0 2024-03-25T18:49:37.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.661+0000 7fc50a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:49:37.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.661+0000 7fc50a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:37.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.661+0000 7fc50a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:37.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.661+0000 7fc50a7fc640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:49:37.662 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.662+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc514137a80 con 0x7fc5141430c0 2024-03-25T18:49:37.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.668+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2410677844 0 0) 0x7fc4f8004eb0 con 0x7fc5140a98f0 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.668+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc514139880 con 0x7fc5140a98f0 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.668+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc510001b00 con 0x7fc5140a98f0 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.668+0000 7fc50a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.668+0000 7fc50a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.668+0000 7fc50a7fc640 10 monclient(hunting): dump: 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:37.669 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:37.670 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:37.670 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:37.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.668+0000 7fc50a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:49:37.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3719613284 0 0) 0x7fc514139880 con 0x7fc5140a98f0 2024-03-25T18:49:37.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 >> v1:172.21.15.32:6790/0 conn(0x7fc5141430c0 legacy=0x7fc5141e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:37.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 >> v1:172.21.15.32:6789/0 conn(0x7fc51413f710 legacy=0x7fc514139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:37.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:49:37.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 1 monclient: found mon.b 2024-03-25T18:49:37.675 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc514139a10 con 0x7fc5140a98f0 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc50a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 5 monclient: authenticate success, global_id 4681 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 10 monclient: _renew_subs 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5141dab00 con 0x7fc5140a98f0 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 10 client.4681.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 10 monclient: _renew_subs 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:37.676 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc51413b4f0 con 0x7fc5140a98f0 2024-03-25T18:49:37.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.674+0000 7fc51bd5d640 1 librados: init done 2024-03-25T18:49:37.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.676+0000 7fc51bd5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:37.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.676+0000 7fc51bd5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:37.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.676+0000 7fc51bd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:37.677 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.676+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5141dae00 con 0x7fc5140a98f0 2024-03-25T18:49:37.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.968+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc510001d20 con 0x7fc5140a98f0 2024-03-25T18:49:37.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.968+0000 7fc50a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:37.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.968+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fc510004d90 con 0x7fc5140a98f0 2024-03-25T18:49:37.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.968+0000 7fc50a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:37.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.968+0000 7fc50a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:37.968 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.968+0000 7fc50a7fc640 10 monclient: dump: 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:37.969 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:37.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.968+0000 7fc50bfff640 4 set_mon_vals no callback set 2024-03-25T18:49:37.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.969+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fc51001ad60 con 0x7fc5140a98f0 2024-03-25T18:49:37.970 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.969+0000 7fc5192f8640 10 monclient: get_auth_request con 0x7fc4f806dd50 auth_method 0 2024-03-25T18:49:37.971 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.970+0000 7fc50a7fc640 10 client.4681.objecter ms_handle_connect 0x7fc4f806dd50 2024-03-25T18:49:37.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.974+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7fc510054950 con 0x7fc5140a98f0 2024-03-25T18:49:37.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.974+0000 7fc50a7fc640 10 client.4681.objecter ms_dispatch 0x7fc514000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:49:37.974 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.974+0000 7fc50a7fc640 3 client.4681.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:49:37.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.974+0000 7fc50a7fc640 3 client.4681.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:49:37.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.974+0000 7fc50a7fc640 20 client.4681.objecter dump_active .. 0 homeless 2024-03-25T18:49:37.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.974+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc510050290 con 0x7fc5140a98f0 2024-03-25T18:49:37.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.974+0000 7fc50a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:37.975 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:37.974+0000 7fc50a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:49:38.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.233+0000 7fc51bd5d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:38.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.233+0000 7fc51bd5d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:38.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.233+0000 7fc51bd5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:38.234 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.233+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5140b3e80 con 0x7fc5140a98f0 2024-03-25T18:49:38.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.255+0000 7fc50a7fc640 1 -- 172.21.15.32:0/170863849 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7fc510052390 con 0x7fc5140a98f0 2024-03-25T18:49:38.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.255+0000 7fc50a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:38.255 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.255+0000 7fc50a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:49:38.256 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:49:38.256 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:49:38.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 10 librados: watch_flush enter 2024-03-25T18:49:38.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 10 librados: watch_flush exit 2024-03-25T18:49:38.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 20 client.4681.objecter shutdown clearing up homeless session... 2024-03-25T18:49:38.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 10 client.4681.objecter successfully canceled tick 2024-03-25T18:49:38.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 >> v1:172.21.15.136:6800/1141502808 conn(0x7fc4f806dd50 legacy=0x7fc4f8070210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:38.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 10 monclient: shutdown 2024-03-25T18:49:38.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:38.258 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 >> v1:172.21.15.136:6789/0 conn(0x7fc5140a98f0 legacy=0x7fc514137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:38.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 shutdown_connections 2024-03-25T18:49:38.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 >> 172.21.15.32:0/170863849 conn(0x7fc5140b7ea0 msgr2=0x7fc514074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:38.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 shutdown_connections 2024-03-25T18:49:38.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 1 -- 172.21.15.32:0/170863849 wait complete. 2024-03-25T18:49:38.259 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:38.257+0000 7fc51bd5d640 1 librados: shutdown 2024-03-25T18:49:38.268 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:49:45.074 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:49:45.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.187+0000 7fe9e32f6640 10 monclient: get_monmap_and_config 2024-03-25T18:49:45.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.187+0000 7fe9e32f6640 10 monclient: build_initial_monmap 2024-03-25T18:49:45.187 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.187+0000 7fe9e32f6640 10 monclient: monmap: 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:49:45.188176+0000 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:49:45.188176+0000 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:49:45.188 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:45.189 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.188+0000 7fe9e32f6640 1 Processor -- start 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.188+0000 7fe9e32f6640 1 -- start start 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.188+0000 7fe9e32f6640 10 monclient: init 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9dc0a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9dc13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9dc1430c0 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc142520 con 0x7fe9dc0a98f0 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc0b28a0 con 0x7fe9dc13f710 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc0b2a30 con 0x7fe9dc1430c0 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:45.190 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.189+0000 7fe9e32f6640 10 monclient(hunting): authenticate will time out at 6691.378321s 2024-03-25T18:49:45.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.195+0000 7fe9e0891640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fe9dc13f710 0x7fe9dc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:36556/0 (socket says 172.21.15.32:36556) 2024-03-25T18:49:45.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.195+0000 7fe9e0891640 1 -- 172.21.15.32:0/1246372404 learned_addr learned my addr 172.21.15.32:0/1246372404 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:45.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.208+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3904593854 0 0) 0x7fe9dc0b28a0 con 0x7fe9dc13f710 2024-03-25T18:49:45.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.208+0000 7fe9d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:49:45.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.208+0000 7fe9d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:45.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.208+0000 7fe9d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:45.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.208+0000 7fe9d3fff640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:49:45.209 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.208+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c8003520 con 0x7fe9dc13f710 2024-03-25T18:49:45.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.213+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930488879 0 0) 0x7fe9dc0b2a30 con 0x7fe9dc1430c0 2024-03-25T18:49:45.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.213+0000 7fe9d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:49:45.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.213+0000 7fe9d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:45.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.213+0000 7fe9d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:45.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.213+0000 7fe9d3fff640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:49:45.213 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.213+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9dc0b28a0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3903977462 0 0) 0x7fe9c8003520 con 0x7fe9dc13f710 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 >> v1:172.21.15.32:6790/0 conn(0x7fe9dc1430c0 legacy=0x7fe9dc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 >> v1:172.21.15.32:6789/0 conn(0x7fe9dc0a98f0 legacy=0x7fe9dc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 1 monclient: found mon.noname-c 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc0b44c0 con 0x7fe9dc13f710 2024-03-25T18:49:45.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe9cc001690 con 0x7fe9dc13f710 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9d3fff640 10 monclient: dump: 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:45.224 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:45.225 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:45.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9e32f6640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:49:45.225 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.223+0000 7fe9e32f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.237+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9cc0017f0 con 0x7fe9dc13f710 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.237+0000 7fe9d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.237+0000 7fe9d3fff640 1 -- 172.21.15.32:0/1246372404 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe9cc001c00 con 0x7fe9dc13f710 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.237+0000 7fe9d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.237+0000 7fe9d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.237+0000 7fe9d3fff640 10 monclient: dump: 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:45.237 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:45.238 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:45.238 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:45.238 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:45.238 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:45.238 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:45.238 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:45.238 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.237+0000 7fe9e32f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:49:45.238 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.237+0000 7fe9e32f6640 4 set_mon_vals no callback set 2024-03-25T18:49:45.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.238+0000 7fe9e32f6640 10 monclient: shutdown 2024-03-25T18:49:45.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.238+0000 7fe9e32f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:45.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.238+0000 7fe9e32f6640 1 -- 172.21.15.32:0/1246372404 >> v1:172.21.15.136:6789/0 conn(0x7fe9dc13f710 legacy=0x7fe9dc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:45.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.238+0000 7fe9e32f6640 1 -- 172.21.15.32:0/1246372404 shutdown_connections 2024-03-25T18:49:45.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.238+0000 7fe9e32f6640 1 -- 172.21.15.32:0/1246372404 >> 172.21.15.32:0/1246372404 conn(0x7fe9dc0b7ea0 msgr2=0x7fe9dc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:45.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.238+0000 7fe9e32f6640 1 -- 172.21.15.32:0/1246372404 shutdown_connections 2024-03-25T18:49:45.239 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.238+0000 7fe9e32f6640 1 -- 172.21.15.32:0/1246372404 wait complete. 2024-03-25T18:49:45.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.239+0000 7fe9e32f6640 10 monclient: build_initial_monmap 2024-03-25T18:49:45.240 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.239+0000 7fe9e32f6640 10 monclient: monmap: 2024-03-25T18:49:45.240 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:45.240 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:49:45.240 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:49:45.240 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.240+0000 7fe9e32f6640 1 librados: starting msgr at 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.240+0000 7fe9e32f6640 1 librados: starting objecter 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.240+0000 7fe9e32f6640 1 Processor -- start 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.240+0000 7fe9e32f6640 1 -- start start 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.240+0000 7fe9e32f6640 1 librados: setting wanted keys 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.240+0000 7fe9e32f6640 1 librados: calling monclient init 2024-03-25T18:49:45.241 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.240+0000 7fe9e32f6640 10 monclient: init 2024-03-25T18:49:45.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:45.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:49:45.242 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 10 monclient(hunting): picked mon.noname-c con 0x7fe9dc0a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:45.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 10 monclient(hunting): picked mon.noname-b con 0x7fe9dc13f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:45.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 10 monclient(hunting): picked mon.noname-a con 0x7fe9dc1430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:45.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc06f5b0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc070510 con 0x7fe9dc13f710 2024-03-25T18:49:45.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe9dc0707f0 con 0x7fe9dc0a98f0 2024-03-25T18:49:45.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:45.243 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.241+0000 7fe9e32f6640 10 monclient(hunting): authenticate will time out at 6691.429878s 2024-03-25T18:49:45.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.250+0000 7fe9e1893640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7fe9dc1430c0 0x7fe9dc1dbdc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:52082/0 (socket says 172.21.15.32:52082) 2024-03-25T18:49:45.250 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.250+0000 7fe9e1893640 1 -- 172.21.15.32:0/2981152860 learned_addr learned my addr 172.21.15.32:0/2981152860 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:45.252 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.252+0000 7fe9d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe9dc1430c0 2024-03-25T18:49:45.252 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.252+0000 7fe9d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:45.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.257+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366070825 0 0) 0x7fe9dc06f5b0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.257+0000 7fe9d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:49:45.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.257+0000 7fe9d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:45.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.257+0000 7fe9d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:45.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.257+0000 7fe9d1ffb640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:49:45.257 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.257+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c0004eb0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.262+0000 7fe9d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe9dc13f710 2024-03-25T18:49:45.261 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.262+0000 7fe9d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:45.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.263+0000 7fe9d1ffb640 10 client.?.objecter ms_handle_connect 0x7fe9dc0a98f0 2024-03-25T18:49:45.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.263+0000 7fe9d1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:45.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.268+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3009079008 0 0) 0x7fe9c0004eb0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.268 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.268+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe9dc06f5b0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.268+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe9d4001c70 con 0x7fe9dc1430c0 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.268+0000 7fe9d1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.268+0000 7fe9d1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.268+0000 7fe9d1ffb640 10 monclient(hunting): dump: 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:45.269 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.268+0000 7fe9d1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:49:45.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.274+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4255285723 0 0) 0x7fe9dc070510 con 0x7fe9dc13f710 2024-03-25T18:49:45.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.274+0000 7fe9d1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:49:45.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.274+0000 7fe9d1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:45.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.274+0000 7fe9d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:45.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.274+0000 7fe9d1ffb640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:49:45.274 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.274+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe9c0004eb0 con 0x7fe9dc13f710 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3080198550 0 0) 0x7fe9dc06f5b0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 >> v1:172.21.15.32:6790/0 conn(0x7fe9dc0a98f0 legacy=0x7fe9dc06e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 >> v1:172.21.15.136:6789/0 conn(0x7fe9dc13f710 legacy=0x7fe9dc138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 1 monclient: found mon.a 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9dc06f030 con 0x7fe9dc1430c0 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:45.280 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 5 monclient: authenticate success, global_id 4713 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 10 monclient: _renew_subs 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9dc06f2e0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 10 monclient: _renew_subs 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9dc1398c0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.279+0000 7fe9e32f6640 1 librados: init done 2024-03-25T18:49:45.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.280+0000 7fe9e32f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:45.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.280+0000 7fe9e32f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:45.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.280+0000 7fe9e32f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:45.282 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.280+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9dc13d010 con 0x7fe9dc1430c0 2024-03-25T18:49:45.600 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.600+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe9d4003990 con 0x7fe9dc1430c0 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.600+0000 7fe9d1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.600+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fe9d4003da0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.600+0000 7fe9d1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.600+0000 7fe9d1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.600+0000 7fe9d1ffb640 10 monclient: dump: 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:45.601 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:45.602 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:45.602 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:45.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.600+0000 7fe9d37fe640 4 set_mon_vals no callback set 2024-03-25T18:49:45.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.601+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fe9d4003fc0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.601+0000 7fe9e1092640 10 monclient: get_auth_request con 0x7fe9c006dd90 auth_method 0 2024-03-25T18:49:45.602 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.602+0000 7fe9d1ffb640 10 client.4713.objecter ms_handle_connect 0x7fe9c006dd90 2024-03-25T18:49:45.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.607+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7fe9d4054ab0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.607+0000 7fe9d1ffb640 10 client.4713.objecter ms_dispatch 0x7fe9dc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:49:45.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.607+0000 7fe9d1ffb640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:49:45.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.607+0000 7fe9d1ffb640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:49:45.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.607+0000 7fe9d1ffb640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T18:49:45.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.607+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe9d40505c0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.608+0000 7fe9d1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:45.608 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.608+0000 7fe9d1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:49:45.834 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.834+0000 7fe9e32f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:45.834 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.834+0000 7fe9e32f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:45.835 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.834+0000 7fe9e32f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:49:45.835 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.834+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe9dc0627b0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.859+0000 7fe9d1ffb640 1 -- 172.21.15.32:0/2981152860 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7fe9d40524f0 con 0x7fe9dc1430c0 2024-03-25T18:49:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.859+0000 7fe9d1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:45.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.859+0000 7fe9d1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:49:45.860 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:49:45.860 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 10 librados: watch_flush enter 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 10 librados: watch_flush exit 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 10 client.4713.objecter successfully canceled tick 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 >> v1:172.21.15.136:6800/1141502808 conn(0x7fe9c006dd90 legacy=0x7fe9c0070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 10 monclient: shutdown 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 >> v1:172.21.15.32:6789/0 conn(0x7fe9dc1430c0 legacy=0x7fe9dc1dbdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 shutdown_connections 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.862+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 >> 172.21.15.32:0/2981152860 conn(0x7fe9dc0b7ea0 msgr2=0x7fe9dc14e3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.863+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 shutdown_connections 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.863+0000 7fe9e32f6640 1 -- 172.21.15.32:0/2981152860 wait complete. 2024-03-25T18:49:45.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:45.863+0000 7fe9e32f6640 1 librados: shutdown 2024-03-25T18:49:45.875 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:49:52.681 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:49:52.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.796+0000 7f4a0c510640 10 monclient: get_monmap_and_config 2024-03-25T18:49:52.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.796+0000 7f4a0c510640 10 monclient: build_initial_monmap 2024-03-25T18:49:52.797 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.797+0000 7f4a0c510640 10 monclient: monmap: 2024-03-25T18:49:52.797 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:52.797 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:49:52.797459+0000 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:49:52.797459+0000 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.797+0000 7f4a0c510640 1 Processor -- start 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.797+0000 7f4a0c510640 1 -- start start 2024-03-25T18:49:52.798 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.797+0000 7f4a0c510640 10 monclient: init 2024-03-25T18:49:52.799 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.798+0000 7f4a0c510640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.798+0000 7f4a0c510640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.799+0000 7f4a0c510640 10 monclient(hunting): picked mon.noname-a con 0x7f4a040a98f0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.799+0000 7f4a0c510640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.799+0000 7f4a0c510640 10 monclient(hunting): picked mon.noname-c con 0x7f4a041430c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.799+0000 7f4a0c510640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a04142520 con 0x7f4a040a98f0 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.799+0000 7f4a0c510640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a040b28a0 con 0x7f4a041430c0 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.799+0000 7f4a0c510640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a040b2a30 con 0x7f4a0413f710 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.799+0000 7f4a0c510640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:52.800 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.799+0000 7f4a0c510640 10 monclient(hunting): authenticate will time out at 6698.988215s 2024-03-25T18:49:52.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.806+0000 7f4a0aaad640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4a041430c0 0x7f4a0414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:49566/0 (socket says 172.21.15.32:49566) 2024-03-25T18:49:52.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.806+0000 7f4a0aaad640 1 -- 172.21.15.32:0/2249676462 learned_addr learned my addr 172.21.15.32:0/2249676462 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:52.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.821+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 792847355 0 0) 0x7f4a040b28a0 con 0x7f4a041430c0 2024-03-25T18:49:52.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.821+0000 7f4a092aa640 10 monclient(hunting): _negotiate 2024-03-25T18:49:52.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.821+0000 7f4a092aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:52.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.821+0000 7f4a092aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:52.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.821+0000 7f4a092aa640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:49:52.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.821+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49f0003520 con 0x7f4a041430c0 2024-03-25T18:49:52.826 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.826+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359473967 0 0) 0x7f4a04142520 con 0x7f4a040a98f0 2024-03-25T18:49:52.826 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.826+0000 7f4a092aa640 10 monclient(hunting): _negotiate 2024-03-25T18:49:52.826 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.826+0000 7f4a092aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:52.826 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.826+0000 7f4a092aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:52.826 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.826+0000 7f4a092aa640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:49:52.826 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.826+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a040b28a0 con 0x7f4a040a98f0 2024-03-25T18:49:52.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.839+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 642602545 0 0) 0x7f4a040b2a30 con 0x7f4a0413f710 2024-03-25T18:49:52.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.839+0000 7f4a092aa640 10 monclient(hunting): _negotiate 2024-03-25T18:49:52.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.839+0000 7f4a092aa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:52.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.839+0000 7f4a092aa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:52.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.839+0000 7f4a092aa640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:49:52.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.839+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a04142520 con 0x7f4a0413f710 2024-03-25T18:49:52.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 285775807 0 0) 0x7f49f0003520 con 0x7f4a041430c0 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 >> v1:172.21.15.32:6790/0 conn(0x7f4a0413f710 legacy=0x7f4a041429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 >> v1:172.21.15.32:6789/0 conn(0x7f4a040a98f0 legacy=0x7f4a040a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 10 monclient: _finish_hunting 0 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 1 monclient: found mon.noname-c 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a040b44c0 con 0x7f4a041430c0 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 10 monclient: _finish_auth 0 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 10 monclient: _check_auth_tickets 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f49fc001a10 con 0x7f4a041430c0 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a0c510640 5 monclient: authenticate success, global_id 4690 2024-03-25T18:49:52.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a0c510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a092aa640 10 monclient: dump: 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:52.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.841+0000 7f4a0c510640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:49:52.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.856+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49fc001b70 con 0x7f4a041430c0 2024-03-25T18:49:52.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.856+0000 7f4a092aa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:52.856 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.856+0000 7f4a092aa640 1 -- 172.21.15.32:0/2249676462 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f49fc001f80 con 0x7f4a041430c0 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.856+0000 7f4a092aa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.856+0000 7f4a092aa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.856+0000 7f4a092aa640 10 monclient: dump: 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:52.857 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:52.858 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:52.858 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:52.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.856+0000 7f4a0c510640 10 monclient: get_monmap_and_config success 2024-03-25T18:49:52.858 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.856+0000 7f4a0c510640 4 set_mon_vals no callback set 2024-03-25T18:49:52.859 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.857+0000 7f4a0c510640 10 monclient: shutdown 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.857+0000 7f4a0c510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.857+0000 7f4a0c510640 1 -- 172.21.15.32:0/2249676462 >> v1:172.21.15.136:6789/0 conn(0x7f4a041430c0 legacy=0x7f4a0414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.857+0000 7f4a0c510640 1 -- 172.21.15.32:0/2249676462 shutdown_connections 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.857+0000 7f4a0c510640 1 -- 172.21.15.32:0/2249676462 >> 172.21.15.32:0/2249676462 conn(0x7f4a040b7ea0 msgr2=0x7f4a040b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.857+0000 7f4a0c510640 1 -- 172.21.15.32:0/2249676462 shutdown_connections 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.858+0000 7f4a0c510640 1 -- 172.21.15.32:0/2249676462 wait complete. 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.858+0000 7f4a0c510640 10 monclient: build_initial_monmap 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.858+0000 7f4a0c510640 10 monclient: monmap: 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:49:52.860 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.859+0000 7f4a0c510640 1 librados: starting msgr at 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.859+0000 7f4a0c510640 1 librados: starting objecter 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.859+0000 7f4a0c510640 1 Processor -- start 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.859+0000 7f4a0c510640 1 -- start start 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.859+0000 7f4a0c510640 1 librados: setting wanted keys 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.859+0000 7f4a0c510640 1 librados: calling monclient init 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.859+0000 7f4a0c510640 10 monclient: init 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 10 monclient: _reopen_session rank -1 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:49:52.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 10 monclient(hunting): picked mon.noname-c con 0x7f4a040a98f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:49:52.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 10 monclient(hunting): picked mon.noname-b con 0x7f4a0413f710 addr v1:172.21.15.136:6789/0 2024-03-25T18:49:52.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 10 monclient(hunting): picked mon.noname-a con 0x7f4a041430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:49:52.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a041369d0 con 0x7f4a041430c0 2024-03-25T18:49:52.862 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a04133a60 con 0x7f4a0413f710 2024-03-25T18:49:52.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a040ab9e0 con 0x7f4a040a98f0 2024-03-25T18:49:52.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.860+0000 7f4a0c510640 10 monclient(hunting): _renew_subs 2024-03-25T18:49:52.863 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.861+0000 7f4a0c510640 10 monclient(hunting): authenticate will time out at 6699.049572s 2024-03-25T18:49:52.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.869+0000 7f4a09aab640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f4a0413f710 0x7f4a041362c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:49622/0 (socket says 172.21.15.32:49622) 2024-03-25T18:49:52.869 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.869+0000 7f4a09aab640 1 -- 172.21.15.32:0/2846742794 learned_addr learned my addr 172.21.15.32:0/2846742794 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:49:52.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.870+0000 7f49faffd640 10 client.?.objecter ms_handle_connect 0x7f4a0413f710 2024-03-25T18:49:52.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.870+0000 7f49faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:52.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.876+0000 7f49faffd640 10 client.?.objecter ms_handle_connect 0x7f4a040a98f0 2024-03-25T18:49:52.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.876+0000 7f49faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:52.878 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.879+0000 7f49faffd640 10 client.?.objecter ms_handle_connect 0x7f4a041430c0 2024-03-25T18:49:52.879 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.879+0000 7f49faffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:49:52.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.888+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3004447871 0 0) 0x7f4a04133a60 con 0x7f4a0413f710 2024-03-25T18:49:52.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.888+0000 7f49faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:49:52.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.888+0000 7f49faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:52.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.888+0000 7f49faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:52.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.888+0000 7f49faffd640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:49:52.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.888+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e8004eb0 con 0x7f4a0413f710 2024-03-25T18:49:52.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.902+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102588725 0 0) 0x7f4a040ab9e0 con 0x7f4a040a98f0 2024-03-25T18:49:52.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.902+0000 7f49faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:49:52.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.902+0000 7f49faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:52.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.902+0000 7f49faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:52.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.902+0000 7f49faffd640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:49:52.902 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.902+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a04133a60 con 0x7f4a040a98f0 2024-03-25T18:49:52.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.903+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4032872029 0 0) 0x7f49e8004eb0 con 0x7f4a0413f710 2024-03-25T18:49:52.903 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.903+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a040ab9e0 con 0x7f4a0413f710 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.903+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f49f4001a10 con 0x7f4a0413f710 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.903+0000 7f49faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.903+0000 7f49faffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.903+0000 7f49faffd640 10 monclient(hunting): dump: 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:52.904 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:52.905 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:52.905 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:52.905 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.903+0000 7f49faffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:49:52.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.910+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 299041726 0 0) 0x7f4a041369d0 con 0x7f4a041430c0 2024-03-25T18:49:52.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.910+0000 7f49faffd640 10 monclient(hunting): _negotiate 2024-03-25T18:49:52.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.910+0000 7f49faffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:49:52.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.910+0000 7f49faffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:49:52.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.910+0000 7f49faffd640 10 monclient(hunting): my global_id is 4719 2024-03-25T18:49:52.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.910+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49e8004eb0 con 0x7f4a041430c0 2024-03-25T18:49:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.918+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2584645577 0 0) 0x7f4a04133a60 con 0x7f4a040a98f0 2024-03-25T18:49:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.918+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a041369d0 con 0x7f4a040a98f0 2024-03-25T18:49:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.918+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f4a00001970 con 0x7f4a040a98f0 2024-03-25T18:49:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.918+0000 7f49faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.918+0000 7f49faffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:49:52.918 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.918+0000 7f49faffd640 10 monclient(hunting): dump: 2024-03-25T18:49:52.918 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:52.919 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:52.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4109417734 0 0) 0x7f4a040ab9e0 con 0x7f4a0413f710 2024-03-25T18:49:52.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 >> v1:172.21.15.32:6790/0 conn(0x7f4a040a98f0 legacy=0x7f4a04132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:52.921 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 >> v1:172.21.15.32:6789/0 conn(0x7f4a041430c0 legacy=0x7f4a04133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 10 monclient: _finish_hunting 0 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 1 monclient: found mon.b 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a040abcc0 con 0x7f4a0413f710 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 10 monclient: _finish_auth 0 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 10 monclient: _check_auth_tickets 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f49faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 10 monclient: _renew_subs 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a040abfa0 con 0x7f4a0413f710 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 10 client.4693.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:49:52.922 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 10 monclient: _renew_subs 2024-03-25T18:49:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a04137e40 con 0x7f4a0413f710 2024-03-25T18:49:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.920+0000 7f4a0c510640 1 librados: init done 2024-03-25T18:49:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.921+0000 7f4a0c510640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.921+0000 7f4a0c510640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.921+0000 7f4a0c510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:52.923 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:52.921+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a040627b0 con 0x7f4a0413f710 2024-03-25T18:49:53.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.190+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f49f4001c70 con 0x7f4a0413f710 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.190+0000 7f49faffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.190+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f49f4003a60 con 0x7f4a0413f710 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.190+0000 7f49faffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.190+0000 7f49faffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.190+0000 7f49faffd640 10 monclient: dump: 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:49:53.192 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:49:53.193 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:49:53.193 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:49:53.193 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:49:53.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.190+0000 7f4a08aa9640 4 set_mon_vals no callback set 2024-03-25T18:49:53.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.191+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f49f4003c40 con 0x7f4a0413f710 2024-03-25T18:49:53.193 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.192+0000 7f4a0a2ac640 10 monclient: get_auth_request con 0x7f49e806dfb0 auth_method 0 2024-03-25T18:49:53.194 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.192+0000 7f49faffd640 10 client.4693.objecter ms_handle_connect 0x7f49e806dfb0 2024-03-25T18:49:53.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.197+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7f49f4054880 con 0x7f4a0413f710 2024-03-25T18:49:53.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.197+0000 7f49faffd640 10 client.4693.objecter ms_dispatch 0x7f4a04000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:49:53.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.197+0000 7f49faffd640 3 client.4693.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:49:53.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.198+0000 7f49faffd640 3 client.4693.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:49:53.199 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.198+0000 7f49faffd640 20 client.4693.objecter dump_active .. 0 homeless 2024-03-25T18:49:53.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.199+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f49f4050390 con 0x7f4a0413f710 2024-03-25T18:49:53.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.199+0000 7f49faffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:49:53.200 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.199+0000 7f49faffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:49:53.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.440+0000 7f4a0c510640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:53.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.440+0000 7f4a0c510640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:53.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.440+0000 7f4a0c510640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:49:53.441 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.440+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a0413dfd0 con 0x7f4a0413f710 2024-03-25T18:49:53.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.468+0000 7f49faffd640 1 -- 172.21.15.32:0/2846742794 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7f49f40522c0 con 0x7f4a0413f710 2024-03-25T18:49:53.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.469+0000 7f49faffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:49:53.470 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.469+0000 7f49faffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:49:53.470 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:49:53.470 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:49:53.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 10 librados: watch_flush enter 2024-03-25T18:49:53.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 10 librados: watch_flush exit 2024-03-25T18:49:53.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 20 client.4693.objecter shutdown clearing up homeless session... 2024-03-25T18:49:53.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 10 client.4693.objecter successfully canceled tick 2024-03-25T18:49:53.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 >> v1:172.21.15.136:6800/1141502808 conn(0x7f49e806dfb0 legacy=0x7f49e8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:53.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 10 monclient: shutdown 2024-03-25T18:49:53.473 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:49:53.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 >> v1:172.21.15.136:6789/0 conn(0x7f4a0413f710 legacy=0x7f4a041362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:49:53.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 shutdown_connections 2024-03-25T18:49:53.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.471+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 >> 172.21.15.32:0/2846742794 conn(0x7f4a040b7ea0 msgr2=0x7f4a04074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:49:53.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.472+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 shutdown_connections 2024-03-25T18:49:53.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.472+0000 7f4a0c510640 1 -- 172.21.15.32:0/2846742794 wait complete. 2024-03-25T18:49:53.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:49:53.472+0000 7f4a0c510640 1 librados: shutdown 2024-03-25T18:49:53.484 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:49:54.339 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:06.667 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:54.344 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:06.672 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:54.382 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:06.710 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:54.392 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:06.720 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:54.405 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:06.722 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:54.411 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:06.728 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:54.412 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:06.740 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:54.439 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:06.756 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:49:54.445 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:06.773 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:00.290 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:50:00.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.402+0000 7f2dc2715640 10 monclient: get_monmap_and_config 2024-03-25T18:50:00.403 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.402+0000 7f2dc2715640 10 monclient: build_initial_monmap 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.402+0000 7f2dc2715640 10 monclient: monmap: 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:50:00.404257+0000 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:50:00.404257+0000 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:50:00.404 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:00.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.403+0000 7f2dc2715640 1 Processor -- start 2024-03-25T18:50:00.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.403+0000 7f2dc2715640 1 -- start start 2024-03-25T18:50:00.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.403+0000 7f2dc2715640 10 monclient: init 2024-03-25T18:50:00.405 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 10 monclient(hunting): picked mon.noname-b con 0x7f2dbc0b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 10 monclient(hunting): picked mon.noname-a con 0x7f2dbc0b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 10 monclient(hunting): picked mon.noname-c con 0x7f2dbc14be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0af940 con 0x7f2dbc0b1730 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0b0a70 con 0x7f2dbc14be90 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc0b0d50 con 0x7f2dbc0b4d60 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:00.406 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.404+0000 7f2dc2715640 10 monclient(hunting): authenticate will time out at 6706.594377s 2024-03-25T18:50:00.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.415+0000 7f2dc0cb2640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f2dbc14be90 0x7f2dbc14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43458/0 (socket says 172.21.15.32:43458) 2024-03-25T18:50:00.416 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.415+0000 7f2dc0cb2640 1 -- 172.21.15.32:0/1050049296 learned_addr learned my addr 172.21.15.32:0/1050049296 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:00.514 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.513+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770881274 0 0) 0x7f2dbc0af940 con 0x7f2dbc0b1730 2024-03-25T18:50:00.514 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.513+0000 7f2dbaffd640 10 monclient(hunting): _negotiate 2024-03-25T18:50:00.514 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.513+0000 7f2dbaffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:00.514 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.513+0000 7f2dbaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:00.514 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.513+0000 7f2dbaffd640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:50:00.514 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.513+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da4003520 con 0x7f2dbc0b1730 2024-03-25T18:50:00.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4190545041 0 0) 0x7f2da4003520 con 0x7f2dbc0b1730 2024-03-25T18:50:00.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 >> v1:172.21.15.32:6790/0 conn(0x7f2dbc0b4d60 legacy=0x7f2dbc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:00.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 >> v1:172.21.15.136:6789/0 conn(0x7f2dbc14be90 legacy=0x7f2dbc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:00.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 10 monclient: _finish_hunting 0 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 1 monclient: found mon.noname-a 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc0b1030 con 0x7f2dbc0b1730 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 10 monclient: _finish_auth 0 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 10 monclient: _check_auth_tickets 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2db0001690 con 0x7f2dbc0b1730 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dc2715640 5 monclient: authenticate success, global_id 4722 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dc2715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dbaffd640 10 monclient: dump: 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:00.529 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:00.530 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:00.530 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:00.530 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:00.530 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:00.530 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:00.530 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:00.530 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:00.530 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.527+0000 7f2dc2715640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.547+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2db00017f0 con 0x7f2dbc0b1730 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.547+0000 7f2dbaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.547+0000 7f2dbaffd640 1 -- 172.21.15.32:0/1050049296 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2db0001c00 con 0x7f2dbc0b1730 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.547+0000 7f2dc2715640 10 monclient: get_monmap_and_config success 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.547+0000 7f2dc2715640 4 set_mon_vals no callback set 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.547+0000 7f2dbaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.547+0000 7f2dbaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.547+0000 7f2dbaffd640 10 monclient: dump: 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:00.548 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:00.549 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:00.549 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:00.549 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:00.549 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:00.549 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:00.549 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.548+0000 7f2dc2715640 10 monclient: shutdown 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.548+0000 7f2dc2715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.548+0000 7f2dc2715640 1 -- 172.21.15.32:0/1050049296 >> v1:172.21.15.32:6789/0 conn(0x7f2dbc0b1730 legacy=0x7f2dbc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.548+0000 7f2dc2715640 1 -- 172.21.15.32:0/1050049296 shutdown_connections 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.548+0000 7f2dc2715640 1 -- 172.21.15.32:0/1050049296 >> 172.21.15.32:0/1050049296 conn(0x7f2dbc0617a0 msgr2=0x7f2dbc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.548+0000 7f2dc2715640 1 -- 172.21.15.32:0/1050049296 shutdown_connections 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.548+0000 7f2dc2715640 1 -- 172.21.15.32:0/1050049296 wait complete. 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.549+0000 7f2dc2715640 10 monclient: build_initial_monmap 2024-03-25T18:50:00.550 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.549+0000 7f2dc2715640 10 monclient: monmap: 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:50:00.551 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.549+0000 7f2dc2715640 1 librados: starting msgr at 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.549+0000 7f2dc2715640 1 librados: starting objecter 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.549+0000 7f2dc2715640 1 Processor -- start 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 1 -- start start 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 1 librados: setting wanted keys 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 1 librados: calling monclient init 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 10 monclient: init 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 10 monclient(hunting): picked mon.noname-a con 0x7f2dbc0b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 10 monclient(hunting): picked mon.noname-b con 0x7f2dbc0b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 10 monclient(hunting): picked mon.noname-c con 0x7f2dbc14be90 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc143000 con 0x7f2dbc0b1730 2024-03-25T18:50:00.553 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc138980 con 0x7f2dbc0b4d60 2024-03-25T18:50:00.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2dbc138b10 con 0x7f2dbc14be90 2024-03-25T18:50:00.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:00.554 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.550+0000 7f2dc2715640 10 monclient(hunting): authenticate will time out at 6706.740516s 2024-03-25T18:50:00.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.560+0000 7f2dbbfff640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f2dbc0b1730 0x7f2dbc1db100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:57460/0 (socket says 172.21.15.32:57460) 2024-03-25T18:50:00.561 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.560+0000 7f2dbbfff640 1 -- 172.21.15.32:0/672354099 learned_addr learned my addr 172.21.15.32:0/672354099 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:00.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.565+0000 7f2db8ff9640 10 client.?.objecter ms_handle_connect 0x7f2dbc0b1730 2024-03-25T18:50:00.565 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.565+0000 7f2db8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:00.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.571+0000 7f2db8ff9640 10 client.?.objecter ms_handle_connect 0x7f2dbc0b4d60 2024-03-25T18:50:00.572 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.571+0000 7f2db8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:00.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.584+0000 7f2db8ff9640 10 client.?.objecter ms_handle_connect 0x7f2dbc14be90 2024-03-25T18:50:00.585 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.584+0000 7f2db8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:00.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.588+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1484833958 0 0) 0x7f2dbc143000 con 0x7f2dbc0b1730 2024-03-25T18:50:00.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.588+0000 7f2db8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:50:00.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.588+0000 7f2db8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:00.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.588+0000 7f2db8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:00.589 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.588+0000 7f2db8ff9640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:50:00.590 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.588+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2da8004eb0 con 0x7f2dbc0b1730 2024-03-25T18:50:00.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.665+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228712058 0 0) 0x7f2dbc138b10 con 0x7f2dbc14be90 2024-03-25T18:50:00.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.665+0000 7f2db8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:50:00.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.665+0000 7f2db8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:00.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.665+0000 7f2db8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:00.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.665+0000 7f2db8ff9640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:50:00.666 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.665+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2dbc143000 con 0x7f2dbc14be90 2024-03-25T18:50:00.668 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.667+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 21500187 0 0) 0x7f2da8004eb0 con 0x7f2dbc0b1730 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.667+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2dbc138b10 con 0x7f2dbc0b1730 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.667+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2dac0019d0 con 0x7f2dbc0b1730 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.667+0000 7f2db8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.667+0000 7f2db8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.667+0000 7f2db8ff9640 10 monclient(hunting): dump: 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:00.669 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:00.670 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:00.670 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:00.670 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.667+0000 7f2db8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3133109450 0 0) 0x7f2dbc138b10 con 0x7f2dbc0b1730 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 >> v1:172.21.15.32:6790/0 conn(0x7f2dbc14be90 legacy=0x7f2dbc1e2220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 >> v1:172.21.15.136:6789/0 conn(0x7f2dbc0b4d60 legacy=0x7f2dbc1db810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 1 monclient: found mon.a 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2dbc136160 con 0x7f2dbc0b1730 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2db8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:00.681 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:50:00.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 10 monclient: _renew_subs 2024-03-25T18:50:00.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:00.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2dbc1363b0 con 0x7f2dbc0b1730 2024-03-25T18:50:00.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 10 client.4725.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:50:00.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 10 monclient: _renew_subs 2024-03-25T18:50:00.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:00.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2dbc142af0 con 0x7f2dbc0b1730 2024-03-25T18:50:00.682 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.679+0000 7f2dc2715640 1 librados: init done 2024-03-25T18:50:00.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.681+0000 7f2dc2715640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:00.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.681+0000 7f2dc2715640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:00.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.681+0000 7f2dc2715640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:00.683 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:00.681+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2dbc137340 con 0x7f2dbc0b1730 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.011+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2dac0035f0 con 0x7f2dbc0b1730 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.012+0000 7f2db8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.012+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f2dac003860 con 0x7f2dbc0b1730 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.012+0000 7f2db8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.012+0000 7f2db8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.012+0000 7f2db8ff9640 10 monclient: dump: 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:01.013 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:01.014 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:01.014 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:01.014 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:01.014 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:01.014 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:01.014 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:01.014 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:01.014 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.012+0000 7f2dba7fc640 4 set_mon_vals no callback set 2024-03-25T18:50:01.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.013+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f2dac003a80 con 0x7f2dbc0b1730 2024-03-25T18:50:01.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.013+0000 7f2dbb7fe640 10 monclient: get_auth_request con 0x7f2da806dd10 auth_method 0 2024-03-25T18:50:01.015 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.014+0000 7f2db8ff9640 10 client.4725.objecter ms_handle_connect 0x7f2da806dd10 2024-03-25T18:50:01.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.023+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7f2dac004480 con 0x7f2dbc0b1730 2024-03-25T18:50:01.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.023+0000 7f2db8ff9640 10 client.4725.objecter ms_dispatch 0x7f2dbc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:50:01.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.023+0000 7f2db8ff9640 3 client.4725.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:50:01.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.023+0000 7f2db8ff9640 3 client.4725.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:50:01.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.023+0000 7f2db8ff9640 20 client.4725.objecter dump_active .. 0 homeless 2024-03-25T18:50:01.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.023+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2dac050150 con 0x7f2dbc0b1730 2024-03-25T18:50:01.024 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.023+0000 7f2db8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:01.025 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.023+0000 7f2db8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:50:01.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.274+0000 7f2dc2715640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:01.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.274+0000 7f2dc2715640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:01.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.274+0000 7f2dc2715640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:01.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.274+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2dbc0627b0 con 0x7f2dbc0b1730 2024-03-25T18:50:01.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.329+0000 7f2db8ff9640 1 -- 172.21.15.32:0/672354099 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7f2dac052080 con 0x7f2dbc0b1730 2024-03-25T18:50:01.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.329+0000 7f2db8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:01.330 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.329+0000 7f2db8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:50:01.330 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:50:01.331 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:50:01.333 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.331+0000 7f2dc2715640 10 librados: watch_flush enter 2024-03-25T18:50:01.333 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.331+0000 7f2dc2715640 10 librados: watch_flush exit 2024-03-25T18:50:01.333 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.331+0000 7f2dc2715640 20 client.4725.objecter shutdown clearing up homeless session... 2024-03-25T18:50:01.333 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.331+0000 7f2dc2715640 10 client.4725.objecter successfully canceled tick 2024-03-25T18:50:01.333 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.331+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 >> v1:172.21.15.136:6800/1141502808 conn(0x7f2da806dd10 legacy=0x7f2da80701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:01.333 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.331+0000 7f2dc2715640 10 monclient: shutdown 2024-03-25T18:50:01.333 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.331+0000 7f2dc2715640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:01.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.331+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 >> v1:172.21.15.32:6789/0 conn(0x7f2dbc0b1730 legacy=0x7f2dbc1db100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:01.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.332+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 shutdown_connections 2024-03-25T18:50:01.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.332+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 >> 172.21.15.32:0/672354099 conn(0x7f2dbc0617a0 msgr2=0x7f2dbc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:01.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.332+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 shutdown_connections 2024-03-25T18:50:01.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.332+0000 7f2dc2715640 1 -- 172.21.15.32:0/672354099 wait complete. 2024-03-25T18:50:01.334 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:01.332+0000 7f2dc2715640 1 librados: shutdown 2024-03-25T18:50:01.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:50:08.154 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:50:08.272 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.271+0000 7faa7eee4640 10 monclient: get_monmap_and_config 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.271+0000 7faa7eee4640 10 monclient: build_initial_monmap 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.271+0000 7faa7eee4640 10 monclient: monmap: 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:50:08.273220+0000 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:50:08.273220+0000 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:50:08.273 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:50:08.274 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:08.275 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.272+0000 7faa7eee4640 1 Processor -- start 2024-03-25T18:50:08.275 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.272+0000 7faa7eee4640 1 -- start start 2024-03-25T18:50:08.275 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.272+0000 7faa7eee4640 10 monclient: init 2024-03-25T18:50:08.275 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:08.275 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:50:08.275 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 10 monclient(hunting): picked mon.noname-c con 0x7faa78061830 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:08.275 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 10 monclient(hunting): picked mon.noname-a con 0x7faa7813c8c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:08.275 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 10 monclient(hunting): picked mon.noname-b con 0x7faa78072aa0 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:08.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa78071d40 con 0x7faa7813c8c0 2024-03-25T18:50:08.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa78072ef0 con 0x7faa78061830 2024-03-25T18:50:08.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa78071f70 con 0x7faa78072aa0 2024-03-25T18:50:08.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:08.276 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.273+0000 7faa7eee4640 10 monclient(hunting): authenticate will time out at 6714.462954s 2024-03-25T18:50:08.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.280+0000 7faa7d481640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7faa78072aa0 0x7faa7814a8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:47890/0 (socket says 172.21.15.32:47890) 2024-03-25T18:50:08.281 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.280+0000 7faa7d481640 1 -- 172.21.15.32:0/3886619701 learned_addr learned my addr 172.21.15.32:0/3886619701 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:08.293 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.292+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 99588899 0 0) 0x7faa78071f70 con 0x7faa78072aa0 2024-03-25T18:50:08.293 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.292+0000 7faa777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:50:08.293 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.292+0000 7faa777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:08.293 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.292+0000 7faa777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:08.293 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.292+0000 7faa777fe640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:50:08.293 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.292+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa68003520 con 0x7faa78072aa0 2024-03-25T18:50:08.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.313+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805612490 0 0) 0x7faa78071d40 con 0x7faa7813c8c0 2024-03-25T18:50:08.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.313+0000 7faa777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:50:08.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.313+0000 7faa777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:08.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.313+0000 7faa777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:08.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.313+0000 7faa777fe640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:50:08.314 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.313+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa78071f70 con 0x7faa7813c8c0 2024-03-25T18:50:08.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.395+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 631117047 0 0) 0x7faa78071f70 con 0x7faa7813c8c0 2024-03-25T18:50:08.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.395+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 >> v1:172.21.15.32:6790/0 conn(0x7faa78072aa0 legacy=0x7faa7814a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:08.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.395+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 >> v1:172.21.15.136:6789/0 conn(0x7faa78061830 legacy=0x7faa7813c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:08.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.395+0000 7faa777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:50:08.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.395+0000 7faa777fe640 1 monclient: found mon.noname-a 2024-03-25T18:50:08.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.395+0000 7faa777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:08.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.395+0000 7faa777fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.32:6789/0 2024-03-25T18:50:08.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.395+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7806fbb0 con 0x7faa7813c8c0 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa777fe640 10 monclient: _finish_auth 0 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7faa64001670 con 0x7faa7813c8c0 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa777fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa777fe640 10 monclient: dump: 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:08.398 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:08.399 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:08.399 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:08.399 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:08.399 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:08.399 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:08.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa7eee4640 5 monclient: authenticate success, global_id 4728 2024-03-25T18:50:08.399 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.396+0000 7faa7eee4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:08.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.422+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 <== mon.0 v1:172.21.15.32:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa640017d0 con 0x7faa7813c8c0 2024-03-25T18:50:08.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.422+0000 7faa777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:08.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.422+0000 7faa777fe640 1 -- 172.21.15.32:0/3886619701 <== mon.0 v1:172.21.15.32:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7faa64001ba0 con 0x7faa7813c8c0 2024-03-25T18:50:08.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.422+0000 7faa7eee4640 10 monclient: get_monmap_and_config success 2024-03-25T18:50:08.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.422+0000 7faa7eee4640 4 set_mon_vals no callback set 2024-03-25T18:50:08.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.422+0000 7faa777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:08.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.422+0000 7faa777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:50:08.423 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.422+0000 7faa777fe640 10 monclient: dump: 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:08.424 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:08.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.423+0000 7faa7eee4640 10 monclient: shutdown 2024-03-25T18:50:08.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.423+0000 7faa7eee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:08.426 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.423+0000 7faa7eee4640 1 -- 172.21.15.32:0/3886619701 >> v1:172.21.15.32:6789/0 conn(0x7faa7813c8c0 legacy=0x7faa78072390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.423+0000 7faa7eee4640 1 -- 172.21.15.32:0/3886619701 shutdown_connections 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.423+0000 7faa7eee4640 1 -- 172.21.15.32:0/3886619701 >> 172.21.15.32:0/3886619701 conn(0x7faa780b0ef0 msgr2=0x7faa780b1300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.423+0000 7faa7eee4640 1 -- 172.21.15.32:0/3886619701 shutdown_connections 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.423+0000 7faa7eee4640 1 -- 172.21.15.32:0/3886619701 wait complete. 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.423+0000 7faa7eee4640 10 monclient: build_initial_monmap 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.424+0000 7faa7eee4640 10 monclient: monmap: 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:08.427 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.424+0000 7faa7eee4640 1 librados: starting msgr at 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.424+0000 7faa7eee4640 1 librados: starting objecter 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.424+0000 7faa7eee4640 1 Processor -- start 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.424+0000 7faa7eee4640 1 -- start start 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.424+0000 7faa7eee4640 1 librados: setting wanted keys 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.424+0000 7faa7eee4640 1 librados: calling monclient init 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.424+0000 7faa7eee4640 10 monclient: init 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 10 monclient(hunting): picked mon.noname-a con 0x7faa78061830 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 10 monclient(hunting): picked mon.noname-c con 0x7faa78072aa0 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 10 monclient(hunting): picked mon.noname-b con 0x7faa7813c8c0 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:08.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa781c8850 con 0x7faa78061830 2024-03-25T18:50:08.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa781c8b30 con 0x7faa7813c8c0 2024-03-25T18:50:08.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa7813da90 con 0x7faa78072aa0 2024-03-25T18:50:08.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:08.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.425+0000 7faa7eee4640 10 monclient(hunting): authenticate will time out at 6714.615071s 2024-03-25T18:50:08.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.437+0000 7faa7d481640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7faa7813c8c0 0x7faa781e2a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:56632/0 (socket says 172.21.15.32:56632) 2024-03-25T18:50:08.438 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.437+0000 7faa7d481640 1 -- 172.21.15.32:0/2961006565 learned_addr learned my addr 172.21.15.32:0/2961006565 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:08.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.439+0000 7faa757fa640 10 client.?.objecter ms_handle_connect 0x7faa7813c8c0 2024-03-25T18:50:08.440 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.439+0000 7faa757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:08.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.457+0000 7faa757fa640 10 client.?.objecter ms_handle_connect 0x7faa78061830 2024-03-25T18:50:08.458 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.457+0000 7faa757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:08.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.459+0000 7faa757fa640 10 client.?.objecter ms_handle_connect 0x7faa78072aa0 2024-03-25T18:50:08.460 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.459+0000 7faa757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:08.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.484+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3116402657 0 0) 0x7faa781c8850 con 0x7faa78061830 2024-03-25T18:50:08.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.484+0000 7faa757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:50:08.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.484+0000 7faa757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:08.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.484+0000 7faa757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:08.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.484+0000 7faa757fa640 10 monclient(hunting): my global_id is 4731 2024-03-25T18:50:08.485 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.484+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5c004eb0 con 0x7faa78061830 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.543+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3093157606 0 0) 0x7faa781c8b30 con 0x7faa7813c8c0 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.543+0000 7faa757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.543+0000 7faa757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.543+0000 7faa757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.543+0000 7faa757fa640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.543+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa781c8850 con 0x7faa7813c8c0 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.544+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1971838763 0 0) 0x7faa5c004eb0 con 0x7faa78061830 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.544+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faa781c8b30 con 0x7faa78061830 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.544+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7faa6c0018c0 con 0x7faa78061830 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.544+0000 7faa757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.544+0000 7faa757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:50:08.545 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.544+0000 7faa757fa640 10 monclient(hunting): dump: 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:08.546 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.544+0000 7faa757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:50:08.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.547+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2624907748 0 0) 0x7faa7813da90 con 0x7faa78072aa0 2024-03-25T18:50:08.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.547+0000 7faa757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:50:08.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.547+0000 7faa757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:08.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.547+0000 7faa757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:08.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.547+0000 7faa757fa640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:50:08.548 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.547+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faa5c004eb0 con 0x7faa78072aa0 2024-03-25T18:50:08.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3681172987 0 0) 0x7faa781c8b30 con 0x7faa78061830 2024-03-25T18:50:08.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 >> v1:172.21.15.32:6790/0 conn(0x7faa78072aa0 legacy=0x7faa781da6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:08.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 >> v1:172.21.15.136:6789/0 conn(0x7faa7813c8c0 legacy=0x7faa781e2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:08.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 10 monclient: _finish_hunting 0 2024-03-25T18:50:08.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 1 monclient: found mon.a 2024-03-25T18:50:08.557 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.32:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa7813dd70 con 0x7faa78061830 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 10 monclient: _finish_auth 0 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 10 monclient: _check_auth_tickets 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 5 monclient: authenticate success, global_id 4731 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 10 monclient: _renew_subs 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.32:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa78133e40 con 0x7faa78061830 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 10 client.4731.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 10 monclient: _renew_subs 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.32:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faa7813cfa0 con 0x7faa78061830 2024-03-25T18:50:08.558 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.556+0000 7faa7eee4640 1 librados: init done 2024-03-25T18:50:08.559 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.557+0000 7faa7eee4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:08.559 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.557+0000 7faa7eee4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:08.559 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.557+0000 7faa7eee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:08.559 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.557+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa780627b0 con 0x7faa78061830 2024-03-25T18:50:08.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.880+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faa6c0035f0 con 0x7faa78061830 2024-03-25T18:50:08.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.880+0000 7faa757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:08.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.880+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7faa6c003a00 con 0x7faa78061830 2024-03-25T18:50:08.881 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.880+0000 7faa757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.880+0000 7faa757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.880+0000 7faa757fa640 10 monclient: dump: 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:08.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.880+0000 7faa76ffd640 4 set_mon_vals no callback set 2024-03-25T18:50:08.883 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.881+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7faa6c003c20 con 0x7faa78061830 2024-03-25T18:50:08.883 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.882+0000 7faa77fff640 10 monclient: get_auth_request con 0x7faa5c06de00 auth_method 0 2024-03-25T18:50:08.883 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.882+0000 7faa757fa640 10 client.4731.objecter ms_handle_connect 0x7faa5c06de00 2024-03-25T18:50:08.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.887+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7faa6c053590 con 0x7faa78061830 2024-03-25T18:50:08.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.887+0000 7faa757fa640 10 client.4731.objecter ms_dispatch 0x7faa78000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:50:08.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.887+0000 7faa757fa640 3 client.4731.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:50:08.888 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.887+0000 7faa757fa640 3 client.4731.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:50:08.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.887+0000 7faa757fa640 20 client.4731.objecter dump_active .. 0 homeless 2024-03-25T18:50:08.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.887+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7faa6c04f0a0 con 0x7faa78061830 2024-03-25T18:50:08.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.887+0000 7faa757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:08.889 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:08.887+0000 7faa757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:50:09.145 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.144+0000 7faa7eee4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:09.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.144+0000 7faa7eee4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:09.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.144+0000 7faa7eee4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.32:6789/0 2024-03-25T18:50:09.146 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.144+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 --> v1:172.21.15.32:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa7813e3a0 con 0x7faa78061830 2024-03-25T18:50:09.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.187+0000 7faa757fa640 1 -- 172.21.15.32:0/2961006565 <== mon.0 v1:172.21.15.32:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7faa6c050fd0 con 0x7faa78061830 2024-03-25T18:50:09.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.187+0000 7faa757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:09.188 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.187+0000 7faa757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:50:09.189 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:50:09.189 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:50:09.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 10 librados: watch_flush enter 2024-03-25T18:50:09.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 10 librados: watch_flush exit 2024-03-25T18:50:09.191 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 20 client.4731.objecter shutdown clearing up homeless session... 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 10 client.4731.objecter successfully canceled tick 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 >> v1:172.21.15.136:6800/1141502808 conn(0x7faa5c06de00 legacy=0x7faa5c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 10 monclient: shutdown 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 >> v1:172.21.15.32:6789/0 conn(0x7faa78061830 legacy=0x7faa78133730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 shutdown_connections 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 >> 172.21.15.32:0/2961006565 conn(0x7faa780b0ef0 msgr2=0x7faa78072350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 shutdown_connections 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 1 -- 172.21.15.32:0/2961006565 wait complete. 2024-03-25T18:50:09.192 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:09.190+0000 7faa7eee4640 1 librados: shutdown 2024-03-25T18:50:09.203 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:50:14.344 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:26.672 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.426 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:26.753 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.459 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:26.787 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.477 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:26.794 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.488 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:26.805 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.489 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:26.817 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.493 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:26.821 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.520 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:26.837 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.524 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:26.852 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.693 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:27.021 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.696 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:27.024 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.752 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:27.069 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.756 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:27.073 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.778 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:27.106 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.778 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:27.106 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.780 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:27.097 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.786 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:27.114 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:14.823 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:27.151 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:16.009 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.119+0000 7f5f6b00f640 10 monclient: get_monmap_and_config 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.119+0000 7f5f6b00f640 10 monclient: build_initial_monmap 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.119+0000 7f5f6b00f640 10 monclient: monmap: 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:50:16.120720+0000 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:50:16.120720+0000 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:16.120 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:16.121 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:50:16.121 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:50:16.121 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 1 Processor -- start 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 1 -- start start 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 10 monclient: init 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6406c380 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6406d670 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 10 monclient(hunting): picked mon.noname-c con 0x7f5f640af8f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f6406c9a0 con 0x7f5f6406c380 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f640ada90 con 0x7f5f640af8f0 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f640b0d10 con 0x7f5f6406d670 2024-03-25T18:50:16.122 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:16.123 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.120+0000 7f5f6b00f640 10 monclient(hunting): authenticate will time out at 6722.310336s 2024-03-25T18:50:16.130 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.129+0000 7f5f68dab640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5f6406c380 0x7f5f640b0600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:43998/0 (socket says 172.21.15.32:43998) 2024-03-25T18:50:16.130 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.129+0000 7f5f68dab640 1 -- 172.21.15.32:0/1126419986 learned_addr learned my addr 172.21.15.32:0/1126419986 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:16.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.147+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 235482768 0 0) 0x7f5f640ada90 con 0x7f5f640af8f0 2024-03-25T18:50:16.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.147+0000 7f5f637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:50:16.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.147+0000 7f5f637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:16.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.147+0000 7f5f637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:16.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.147+0000 7f5f637fe640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:50:16.148 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.147+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f54003520 con 0x7f5f640af8f0 2024-03-25T18:50:16.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.148+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 878681744 0 0) 0x7f5f6406c9a0 con 0x7f5f6406c380 2024-03-25T18:50:16.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.148+0000 7f5f637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:50:16.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.148+0000 7f5f637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:16.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.148+0000 7f5f637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:16.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.148+0000 7f5f637fe640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:50:16.149 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.148+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f640ada90 con 0x7f5f6406c380 2024-03-25T18:50:16.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.155+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1340523278 0 0) 0x7f5f640b0d10 con 0x7f5f6406d670 2024-03-25T18:50:16.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.155+0000 7f5f637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:50:16.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.155+0000 7f5f637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:16.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.155+0000 7f5f637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:16.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.155+0000 7f5f637fe640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:50:16.156 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.155+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f6406c9a0 con 0x7f5f6406d670 2024-03-25T18:50:16.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 613140881 0 0) 0x7f5f54003520 con 0x7f5f640af8f0 2024-03-25T18:50:16.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 >> v1:172.21.15.32:6790/0 conn(0x7f5f6406d670 legacy=0x7f5f640ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:16.159 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 >> v1:172.21.15.32:6789/0 conn(0x7f5f6406c380 legacy=0x7f5f640b0600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 10 monclient: _finish_hunting 0 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 1 monclient: found mon.noname-c 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f640add00 con 0x7f5f640af8f0 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 10 monclient: _finish_auth 0 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 10 monclient: _check_auth_tickets 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f5c001a10 con 0x7f5f640af8f0 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f6b00f640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f6b00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f637fe640 10 monclient: dump: 2024-03-25T18:50:16.160 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:16.161 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.158+0000 7f5f6b00f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:16.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.174+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f5c001b70 con 0x7f5f640af8f0 2024-03-25T18:50:16.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.174+0000 7f5f637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:16.175 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.174+0000 7f5f637fe640 1 -- 172.21.15.32:0/1126419986 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f5c001f00 con 0x7f5f640af8f0 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.174+0000 7f5f6b00f640 10 monclient: get_monmap_and_config success 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.174+0000 7f5f6b00f640 4 set_mon_vals no callback set 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.174+0000 7f5f637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.174+0000 7f5f637fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.174+0000 7f5f637fe640 10 monclient: dump: 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:16.176 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:16.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.175+0000 7f5f6b00f640 10 monclient: shutdown 2024-03-25T18:50:16.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.175+0000 7f5f6b00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:16.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.175+0000 7f5f6b00f640 1 -- 172.21.15.32:0/1126419986 >> v1:172.21.15.136:6789/0 conn(0x7f5f640af8f0 legacy=0x7f5f640aedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:16.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.175+0000 7f5f6b00f640 1 -- 172.21.15.32:0/1126419986 shutdown_connections 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.175+0000 7f5f6b00f640 1 -- 172.21.15.32:0/1126419986 >> 172.21.15.32:0/1126419986 conn(0x7f5f640b5c00 msgr2=0x7f5f640b5fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.175+0000 7f5f6b00f640 1 -- 172.21.15.32:0/1126419986 shutdown_connections 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.175+0000 7f5f6b00f640 1 -- 172.21.15.32:0/1126419986 wait complete. 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 10 monclient: build_initial_monmap 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 10 monclient: monmap: 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:16.179 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 1 librados: starting msgr at 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 1 librados: starting objecter 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 1 Processor -- start 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 1 -- start start 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 1 librados: setting wanted keys 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 1 librados: calling monclient init 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.176+0000 7f5f6b00f640 10 monclient: init 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 10 monclient(hunting): picked mon.noname-a con 0x7f5f6406c380 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 10 monclient(hunting): picked mon.noname-b con 0x7f5f6406d670 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 10 monclient(hunting): picked mon.noname-c con 0x7f5f640af8f0 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:16.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64141e00 con 0x7f5f6406c380 2024-03-25T18:50:16.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64149140 con 0x7f5f6406d670 2024-03-25T18:50:16.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f64149420 con 0x7f5f640af8f0 2024-03-25T18:50:16.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:16.181 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.177+0000 7f5f6b00f640 10 monclient(hunting): authenticate will time out at 6722.367209s 2024-03-25T18:50:16.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.194+0000 7f5f68dab640 1 --1- >> v1:172.21.15.32:6789/0 conn(0x7f5f6406c380 0x7f5f64144040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.32:6789/0 says I am v1:172.21.15.32:44062/0 (socket says 172.21.15.32:44062) 2024-03-25T18:50:16.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.194+0000 7f5f68dab640 1 -- 172.21.15.32:0/4160200184 learned_addr learned my addr 172.21.15.32:0/4160200184 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:16.195 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.195+0000 7f5f617fa640 10 client.?.objecter ms_handle_connect 0x7f5f6406c380 2024-03-25T18:50:16.196 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.195+0000 7f5f617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:16.206 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.205+0000 7f5f617fa640 10 client.?.objecter ms_handle_connect 0x7f5f640af8f0 2024-03-25T18:50:16.206 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.205+0000 7f5f617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:16.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.206+0000 7f5f617fa640 10 client.?.objecter ms_handle_connect 0x7f5f6406d670 2024-03-25T18:50:16.207 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.206+0000 7f5f617fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:16.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.214+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759888876 0 0) 0x7f5f64149140 con 0x7f5f6406d670 2024-03-25T18:50:16.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.214+0000 7f5f617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:50:16.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.214+0000 7f5f617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:16.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.214+0000 7f5f617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:16.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.214+0000 7f5f617fa640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:50:16.215 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.214+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f48004eb0 con 0x7f5f6406d670 2024-03-25T18:50:16.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.221+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2605121487 0 0) 0x7f5f64149420 con 0x7f5f640af8f0 2024-03-25T18:50:16.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.221+0000 7f5f617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:50:16.222 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.221+0000 7f5f617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:16.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.221+0000 7f5f617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:16.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.221+0000 7f5f617fa640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:50:16.223 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.221+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f64149140 con 0x7f5f640af8f0 2024-03-25T18:50:16.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.223+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144608195 0 0) 0x7f5f64141e00 con 0x7f5f6406c380 2024-03-25T18:50:16.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.223+0000 7f5f617fa640 10 monclient(hunting): _negotiate 2024-03-25T18:50:16.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.223+0000 7f5f617fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:16.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.223+0000 7f5f617fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:16.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.223+0000 7f5f617fa640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:50:16.224 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.223+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f64149420 con 0x7f5f6406c380 2024-03-25T18:50:16.232 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.231+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3242149500 0 0) 0x7f5f48004eb0 con 0x7f5f6406d670 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.231+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f64141e00 con 0x7f5f6406d670 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.231+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f500018f0 con 0x7f5f6406d670 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.231+0000 7f5f617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.231+0000 7f5f617fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.231+0000 7f5f617fa640 10 monclient(hunting): dump: 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:16.233 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.231+0000 7f5f617fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:50:16.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.247+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2088966909 0 0) 0x7f5f64149420 con 0x7f5f6406c380 2024-03-25T18:50:16.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.247+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f48004eb0 con 0x7f5f6406c380 2024-03-25T18:50:16.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.247+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f580018f0 con 0x7f5f6406c380 2024-03-25T18:50:16.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.247+0000 7f5f617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:16.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.247+0000 7f5f617fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:50:16.248 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.247+0000 7f5f617fa640 10 monclient(hunting): dump: 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:16.249 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2906840221 0 0) 0x7f5f64149140 con 0x7f5f640af8f0 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5f64149420 con 0x7f5f640af8f0 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f5c001e80 con 0x7f5f640af8f0 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 10 monclient(hunting): dump: 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.263 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 852472011 0 0) 0x7f5f64141e00 con 0x7f5f6406d670 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 >> v1:172.21.15.32:6790/0 conn(0x7f5f640af8f0 legacy=0x7f5f64140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 >> v1:172.21.15.32:6789/0 conn(0x7f5f6406c380 legacy=0x7f5f64144040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 10 monclient: _finish_hunting 0 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 1 monclient: found mon.b 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:16.264 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f64140aa0 con 0x7f5f6406d670 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 10 monclient: _finish_auth 0 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 10 monclient: _check_auth_tickets 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f617fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 10 monclient: _renew_subs 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f64140570 con 0x7f5f6406d670 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 10 client.4711.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 10 monclient: _renew_subs 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f64140820 con 0x7f5f6406d670 2024-03-25T18:50:16.265 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.261+0000 7f5f6b00f640 1 librados: init done 2024-03-25T18:50:16.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.262+0000 7f5f6b00f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:16.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.262+0000 7f5f6b00f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:16.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.262+0000 7f5f6b00f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:16.266 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.263+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f640af480 con 0x7f5f6406d670 2024-03-25T18:50:16.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.536+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5f50003610 con 0x7f5f6406d670 2024-03-25T18:50:16.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.536+0000 7f5f617fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:16.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.536+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f5f50003920 con 0x7f5f6406d670 2024-03-25T18:50:16.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.536+0000 7f5f617fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:16.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.536+0000 7f5f617fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:50:16.537 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.536+0000 7f5f617fa640 10 monclient: dump: 2024-03-25T18:50:16.537 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:16.537 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:16.538 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.536+0000 7f5f62ffd640 4 set_mon_vals no callback set 2024-03-25T18:50:16.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.537+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f5f50019930 con 0x7f5f6406d670 2024-03-25T18:50:16.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.538+0000 7f5f68dab640 10 monclient: get_auth_request con 0x7f5f4806e0f0 auth_method 0 2024-03-25T18:50:16.539 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.538+0000 7f5f617fa640 10 client.4711.objecter ms_handle_connect 0x7f5f4806e0f0 2024-03-25T18:50:16.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.542+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7f5f500535a0 con 0x7f5f6406d670 2024-03-25T18:50:16.543 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.542+0000 7f5f617fa640 10 client.4711.objecter ms_dispatch 0x7f5f64000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:50:16.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.542+0000 7f5f617fa640 3 client.4711.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:50:16.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.542+0000 7f5f617fa640 3 client.4711.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:50:16.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.542+0000 7f5f617fa640 20 client.4711.objecter dump_active .. 0 homeless 2024-03-25T18:50:16.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.543+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5f5004f0b0 con 0x7f5f6406d670 2024-03-25T18:50:16.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.543+0000 7f5f617fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:16.544 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.543+0000 7f5f617fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:50:16.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.795+0000 7f5f6b00f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:16.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.795+0000 7f5f6b00f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:16.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.795+0000 7f5f6b00f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:16.796 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.795+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5f64140c30 con 0x7f5f6406d670 2024-03-25T18:50:16.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.838+0000 7f5f617fa640 1 -- 172.21.15.32:0/4160200184 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7f5f50050fe0 con 0x7f5f6406d670 2024-03-25T18:50:16.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.838+0000 7f5f617fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:16.839 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.838+0000 7f5f617fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:50:16.839 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:50:16.839 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:50:16.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 10 librados: watch_flush enter 2024-03-25T18:50:16.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 10 librados: watch_flush exit 2024-03-25T18:50:16.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 20 client.4711.objecter shutdown clearing up homeless session... 2024-03-25T18:50:16.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 10 client.4711.objecter successfully canceled tick 2024-03-25T18:50:16.842 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 >> v1:172.21.15.136:6800/1141502808 conn(0x7f5f4806e0f0 legacy=0x7f5f480705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:16.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 10 monclient: shutdown 2024-03-25T18:50:16.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:16.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 >> v1:172.21.15.136:6789/0 conn(0x7f5f6406d670 legacy=0x7f5f64149840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:16.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 shutdown_connections 2024-03-25T18:50:16.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 >> 172.21.15.32:0/4160200184 conn(0x7f5f640b5c00 msgr2=0x7f5f6414e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:16.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 shutdown_connections 2024-03-25T18:50:16.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 1 -- 172.21.15.32:0/4160200184 wait complete. 2024-03-25T18:50:16.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:16.841+0000 7f5f6b00f640 1 librados: shutdown 2024-03-25T18:50:16.855 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:50:23.662 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:50:23.777 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.777+0000 7f7fcbb38640 10 monclient: get_monmap_and_config 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.777+0000 7f7fcbb38640 10 monclient: build_initial_monmap 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.777+0000 7f7fcbb38640 10 monclient: monmap: 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:50:23.777533+0000 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:50:23.777533+0000 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:50:23.778 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:23.779 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.777+0000 7f7fcbb38640 1 Processor -- start 2024-03-25T18:50:23.779 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.777+0000 7f7fcbb38640 1 -- start start 2024-03-25T18:50:23.779 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.777+0000 7f7fcbb38640 10 monclient: init 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc406adf0 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc40aee10 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc4140860 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40af690 con 0x7f7fc40aee10 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40b0c70 con 0x7f7fc4140860 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc40b0f50 con 0x7f7fc406adf0 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:23.780 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.778+0000 7f7fcbb38640 10 monclient(hunting): authenticate will time out at 6729.967278s 2024-03-25T18:50:23.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.787+0000 7f7fc98d4640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f7fc406adf0 0x7f7fc413d530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:46724/0 (socket says 172.21.15.32:46724) 2024-03-25T18:50:23.787 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.787+0000 7f7fc98d4640 1 -- 172.21.15.32:0/1765533300 learned_addr learned my addr 172.21.15.32:0/1765533300 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:23.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755737713 0 0) 0x7f7fc40b0c70 con 0x7f7fc4140860 2024-03-25T18:50:23.810 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 10 monclient(hunting): _negotiate 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fb0003520 con 0x7f7fc4140860 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2754634841 0 0) 0x7f7fc40b0f50 con 0x7f7fc406adf0 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 10 monclient(hunting): _negotiate 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:50:23.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.810+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc40b0c70 con 0x7f7fc406adf0 2024-03-25T18:50:23.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.821+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3996753699 0 0) 0x7f7fc40af690 con 0x7f7fc40aee10 2024-03-25T18:50:23.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.821+0000 7f7fc88d2640 10 monclient(hunting): _negotiate 2024-03-25T18:50:23.821 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.821+0000 7f7fc88d2640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:23.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.821+0000 7f7fc88d2640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:23.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.821+0000 7f7fc88d2640 10 monclient(hunting): my global_id is 4740 2024-03-25T18:50:23.822 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.821+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc40b0f50 con 0x7f7fc40aee10 2024-03-25T18:50:23.828 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3420103864 0 0) 0x7f7fc40b0c70 con 0x7f7fc406adf0 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 >> v1:172.21.15.136:6789/0 conn(0x7f7fc4140860 legacy=0x7f7fc414b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 >> v1:172.21.15.32:6789/0 conn(0x7f7fc40aee10 legacy=0x7f7fc4140150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 10 monclient: _finish_hunting 0 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 1 monclient: found mon.noname-b 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.32:6790/0 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 --> v1:172.21.15.32:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc413dc40 con 0x7f7fc406adf0 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 10 monclient: _finish_auth 0 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 10 monclient: _check_auth_tickets 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7fc00019f0 con 0x7f7fc406adf0 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:50:23.829 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fc88d2640 10 monclient: dump: 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fcbb38640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:50:23.830 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.828+0000 7f7fcbb38640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:23.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.840+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 <== mon.2 v1:172.21.15.32:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fc0001b50 con 0x7f7fc406adf0 2024-03-25T18:50:23.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.840+0000 7f7fc88d2640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:23.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.840+0000 7f7fc88d2640 1 -- 172.21.15.32:0/1765533300 <== mon.2 v1:172.21.15.32:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7fc0001f60 con 0x7f7fc406adf0 2024-03-25T18:50:23.840 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.840+0000 7f7fcbb38640 10 monclient: get_monmap_and_config success 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.840+0000 7f7fcbb38640 4 set_mon_vals no callback set 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.840+0000 7f7fc88d2640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.840+0000 7f7fc88d2640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.840+0000 7f7fc88d2640 10 monclient: dump: 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:23.841 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:23.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.841+0000 7f7fcbb38640 10 monclient: shutdown 2024-03-25T18:50:23.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.841+0000 7f7fcbb38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:23.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.841+0000 7f7fcbb38640 1 -- 172.21.15.32:0/1765533300 >> v1:172.21.15.32:6790/0 conn(0x7f7fc406adf0 legacy=0x7f7fc413d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:23.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.841+0000 7f7fcbb38640 1 -- 172.21.15.32:0/1765533300 shutdown_connections 2024-03-25T18:50:23.843 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.841+0000 7f7fcbb38640 1 -- 172.21.15.32:0/1765533300 >> 172.21.15.32:0/1765533300 conn(0x7f7fc40b4a80 msgr2=0x7f7fc40b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.841+0000 7f7fcbb38640 1 -- 172.21.15.32:0/1765533300 shutdown_connections 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.841+0000 7f7fcbb38640 1 -- 172.21.15.32:0/1765533300 wait complete. 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 10 monclient: build_initial_monmap 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 10 monclient: monmap: 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:50:23.844 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 1 librados: starting msgr at 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 1 librados: starting objecter 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 1 Processor -- start 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 1 -- start start 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 1 librados: setting wanted keys 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 1 librados: calling monclient init 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.842+0000 7f7fcbb38640 10 monclient: init 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 10 monclient(hunting): picked mon.noname-a con 0x7f7fc406adf0 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 10 monclient(hunting): picked mon.noname-b con 0x7f7fc40aee10 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 10 monclient(hunting): picked mon.noname-c con 0x7f7fc41440d0 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:23.845 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc41d8870 con 0x7f7fc406adf0 2024-03-25T18:50:23.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc412e4e0 con 0x7f7fc40aee10 2024-03-25T18:50:23.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fc412e7c0 con 0x7f7fc41440d0 2024-03-25T18:50:23.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:23.846 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.843+0000 7f7fcbb38640 10 monclient(hunting): authenticate will time out at 6730.031980s 2024-03-25T18:50:23.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.861+0000 7f7fc90d3640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f7fc40aee10 0x7f7fc41d51b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:55888/0 (socket says 172.21.15.32:55888) 2024-03-25T18:50:23.861 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.861+0000 7f7fc90d3640 1 -- 172.21.15.32:0/3609708955 learned_addr learned my addr 172.21.15.32:0/3609708955 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:23.864 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.864+0000 7f7fba7fc640 10 client.?.objecter ms_handle_connect 0x7f7fc40aee10 2024-03-25T18:50:23.864 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.864+0000 7f7fba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:23.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.868+0000 7f7fba7fc640 10 client.?.objecter ms_handle_connect 0x7f7fc41440d0 2024-03-25T18:50:23.868 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.868+0000 7f7fba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:23.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.871+0000 7f7fba7fc640 10 client.?.objecter ms_handle_connect 0x7f7fc406adf0 2024-03-25T18:50:23.870 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.871+0000 7f7fba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:23.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.876+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845790238 0 0) 0x7f7fc412e4e0 con 0x7f7fc40aee10 2024-03-25T18:50:23.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.876+0000 7f7fba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:50:23.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.876+0000 7f7fba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:23.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.876+0000 7f7fba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:23.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.876+0000 7f7fba7fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:50:23.876 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.876+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fa8004eb0 con 0x7f7fc40aee10 2024-03-25T18:50:23.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.882+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3951029269 0 0) 0x7f7fc412e7c0 con 0x7f7fc41440d0 2024-03-25T18:50:23.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.882+0000 7f7fba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:50:23.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.882+0000 7f7fba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:23.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.882+0000 7f7fba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:23.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.882+0000 7f7fba7fc640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:50:23.882 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.882+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc412e4e0 con 0x7f7fc41440d0 2024-03-25T18:50:23.892 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.892+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 679810065 0 0) 0x7f7fc41d8870 con 0x7f7fc406adf0 2024-03-25T18:50:23.892 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.892+0000 7f7fba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:50:23.892 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.892+0000 7f7fba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:23.892 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.892+0000 7f7fba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:23.892 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.892+0000 7f7fba7fc640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:50:23.892 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.892+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7fc412e7c0 con 0x7f7fc406adf0 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.894+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2037122950 0 0) 0x7f7fc412e4e0 con 0x7f7fc41440d0 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.894+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc41d8870 con 0x7f7fc41440d0 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.894+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7fbc0016f0 con 0x7f7fc41440d0 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.894+0000 7f7fba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.894+0000 7f7fba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.894+0000 7f7fba7fc640 10 monclient(hunting): dump: 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:23.894 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:23.895 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:23.895 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:23.895 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:23.895 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:23.895 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:23.895 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:23.895 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.894+0000 7f7fba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:50:23.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.897+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2696512252 0 0) 0x7f7fa8004eb0 con 0x7f7fc40aee10 2024-03-25T18:50:23.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.897+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7fc412e4e0 con 0x7f7fc40aee10 2024-03-25T18:50:23.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.897+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7fb4001a10 con 0x7f7fc40aee10 2024-03-25T18:50:23.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.897+0000 7f7fba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:23.897 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.897+0000 7f7fba7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.897+0000 7f7fba7fc640 10 monclient(hunting): dump: 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:23.898 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 653838869 0 0) 0x7f7fc412e4e0 con 0x7f7fc40aee10 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 >> v1:172.21.15.32:6790/0 conn(0x7f7fc41440d0 legacy=0x7f7fc41d58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 >> v1:172.21.15.32:6789/0 conn(0x7f7fc406adf0 legacy=0x7f7fc41d8d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 1 monclient: found mon.b 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fc412eaa0 con 0x7f7fc40aee10 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:50:23.909 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 10 monclient: _renew_subs 2024-03-25T18:50:23.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:23.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fc41d9410 con 0x7f7fc40aee10 2024-03-25T18:50:23.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 10 client.4717.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:50:23.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 10 monclient: _renew_subs 2024-03-25T18:50:23.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:23.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fc412cbe0 con 0x7f7fc40aee10 2024-03-25T18:50:23.910 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.908+0000 7f7fcbb38640 1 librados: init done 2024-03-25T18:50:23.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.909+0000 7f7fcbb38640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:23.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.909+0000 7f7fcbb38640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:23.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.909+0000 7f7fcbb38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:23.911 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:23.909+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fc406b1d0 con 0x7f7fc40aee10 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.178+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7fb4001c70 con 0x7f7fc40aee10 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.178+0000 7f7fba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.178+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f7fb4003a60 con 0x7f7fc40aee10 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.178+0000 7f7fba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.178+0000 7f7fba7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.178+0000 7f7fba7fc640 10 monclient: dump: 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:24.178 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:24.179 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:24.179 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:24.179 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:24.179 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:24.179 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:24.179 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:24.179 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:24.179 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.178+0000 7f7fbbfff640 4 set_mon_vals no callback set 2024-03-25T18:50:24.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.179+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f7fb4003c80 con 0x7f7fc40aee10 2024-03-25T18:50:24.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.179+0000 7f7fc98d4640 10 monclient: get_auth_request con 0x7f7fa806dfb0 auth_method 0 2024-03-25T18:50:24.180 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.180+0000 7f7fba7fc640 10 client.4717.objecter ms_handle_connect 0x7f7fa806dfb0 2024-03-25T18:50:24.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.184+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7f7fb4054880 con 0x7f7fc40aee10 2024-03-25T18:50:24.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.184+0000 7f7fba7fc640 10 client.4717.objecter ms_dispatch 0x7f7fc4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:50:24.184 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.184+0000 7f7fba7fc640 3 client.4717.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:50:24.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.184+0000 7f7fba7fc640 3 client.4717.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:50:24.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.184+0000 7f7fba7fc640 20 client.4717.objecter dump_active .. 0 homeless 2024-03-25T18:50:24.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.184+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7fb4050390 con 0x7f7fc40aee10 2024-03-25T18:50:24.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.184+0000 7f7fba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:24.185 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.184+0000 7f7fba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:50:24.409 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.409+0000 7f7fcbb38640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:24.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.409+0000 7f7fcbb38640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:24.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.409+0000 7f7fcbb38640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:24.410 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.409+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fc412cf30 con 0x7f7fc40aee10 2024-03-25T18:50:24.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.450+0000 7f7fba7fc640 1 -- 172.21.15.32:0/3609708955 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7f7fb40522c0 con 0x7f7fc40aee10 2024-03-25T18:50:24.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.450+0000 7f7fba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:24.450 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.450+0000 7f7fba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:50:24.450 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:50:24.451 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 10 librados: watch_flush enter 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 10 librados: watch_flush exit 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 20 client.4717.objecter shutdown clearing up homeless session... 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 10 client.4717.objecter successfully canceled tick 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 >> v1:172.21.15.136:6800/1141502808 conn(0x7f7fa806dfb0 legacy=0x7f7fa8070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 10 monclient: shutdown 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 >> v1:172.21.15.136:6789/0 conn(0x7f7fc40aee10 legacy=0x7f7fc41d51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 shutdown_connections 2024-03-25T18:50:24.453 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 >> 172.21.15.32:0/3609708955 conn(0x7f7fc40b4a80 msgr2=0x7f7fc4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:24.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 shutdown_connections 2024-03-25T18:50:24.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 1 -- 172.21.15.32:0/3609708955 wait complete. 2024-03-25T18:50:24.454 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:24.452+0000 7f7fcbb38640 1 librados: shutdown 2024-03-25T18:50:24.466 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:50:31.272 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:50:31.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.392+0000 7fbf28a4a640 10 monclient: get_monmap_and_config 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.392+0000 7fbf28a4a640 10 monclient: build_initial_monmap 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.392+0000 7fbf28a4a640 10 monclient: monmap: 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:50:31.393392+0000 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:50:31.393392+0000 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:50:31.393 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:31.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.393+0000 7fbf28a4a640 1 Processor -- start 2024-03-25T18:50:31.394 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.393+0000 7fbf28a4a640 1 -- start start 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.393+0000 7fbf28a4a640 10 monclient: init 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fbf240a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fbf2413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fbf241430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf24142520 con 0x7fbf241430c0 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240b28a0 con 0x7fbf240a98f0 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf240b2a30 con 0x7fbf2413f710 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:31.395 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.394+0000 7fbf28a4a640 10 monclient(hunting): authenticate will time out at 6737.583085s 2024-03-25T18:50:31.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.412+0000 7fbf21d9b640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7fbf2413f710 0x7fbf241429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:59484/0 (socket says 172.21.15.32:59484) 2024-03-25T18:50:31.412 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.412+0000 7fbf21d9b640 1 -- 172.21.15.32:0/971723956 learned_addr learned my addr 172.21.15.32:0/971723956 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:31.427 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.427+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1469782259 0 0) 0x7fbf240b2a30 con 0x7fbf2413f710 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf0c003520 con 0x7fbf2413f710 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3036394518 0 0) 0x7fbf240b28a0 con 0x7fbf240a98f0 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:31.428 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:50:31.429 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.428+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf240b2a30 con 0x7fbf240a98f0 2024-03-25T18:50:31.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.436+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 764455397 0 0) 0x7fbf24142520 con 0x7fbf241430c0 2024-03-25T18:50:31.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.436+0000 7fbf2159a640 10 monclient(hunting): _negotiate 2024-03-25T18:50:31.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.436+0000 7fbf2159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:31.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.436+0000 7fbf2159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:31.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.436+0000 7fbf2159a640 10 monclient(hunting): my global_id is 4746 2024-03-25T18:50:31.436 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.436+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf240b28a0 con 0x7fbf241430c0 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3901507600 0 0) 0x7fbf240b2a30 con 0x7fbf240a98f0 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 >> v1:172.21.15.32:6790/0 conn(0x7fbf2413f710 legacy=0x7fbf241429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 >> v1:172.21.15.32:6789/0 conn(0x7fbf241430c0 legacy=0x7fbf2414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 10 monclient: _finish_hunting 0 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 1 monclient: found mon.noname-c 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf240b44c0 con 0x7fbf240a98f0 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 10 monclient: _finish_auth 0 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 10 monclient: _check_auth_tickets 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbf14001670 con 0x7fbf240a98f0 2024-03-25T18:50:31.455 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf2159a640 10 monclient: dump: 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf28a4a640 5 monclient: authenticate success, global_id 4720 2024-03-25T18:50:31.456 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.454+0000 7fbf28a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:31.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.474+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf140017d0 con 0x7fbf240a98f0 2024-03-25T18:50:31.474 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.474+0000 7fbf2159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.474+0000 7fbf2159a640 1 -- 172.21.15.32:0/971723956 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbf14001be0 con 0x7fbf240a98f0 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.474+0000 7fbf2159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.474+0000 7fbf2159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.474+0000 7fbf2159a640 10 monclient: dump: 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:31.477 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:31.478 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:31.478 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:31.478 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:31.478 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:31.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.474+0000 7fbf28a4a640 10 monclient: get_monmap_and_config success 2024-03-25T18:50:31.478 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.474+0000 7fbf28a4a640 4 set_mon_vals no callback set 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.475+0000 7fbf28a4a640 10 monclient: shutdown 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.475+0000 7fbf28a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.475+0000 7fbf28a4a640 1 -- 172.21.15.32:0/971723956 >> v1:172.21.15.136:6789/0 conn(0x7fbf240a98f0 legacy=0x7fbf240a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.475+0000 7fbf28a4a640 1 -- 172.21.15.32:0/971723956 shutdown_connections 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.475+0000 7fbf28a4a640 1 -- 172.21.15.32:0/971723956 >> 172.21.15.32:0/971723956 conn(0x7fbf240b7ea0 msgr2=0x7fbf240b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.475+0000 7fbf28a4a640 1 -- 172.21.15.32:0/971723956 shutdown_connections 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.475+0000 7fbf28a4a640 1 -- 172.21.15.32:0/971723956 wait complete. 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 10 monclient: build_initial_monmap 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 10 monclient: monmap: 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:50:31.480 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 1 librados: starting msgr at 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 1 librados: starting objecter 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 1 Processor -- start 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 1 -- start start 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 1 librados: setting wanted keys 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 1 librados: calling monclient init 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.476+0000 7fbf28a4a640 10 monclient: init 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:50:31.481 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 10 monclient(hunting): picked mon.noname-b con 0x7fbf240a98f0 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:31.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 10 monclient(hunting): picked mon.noname-c con 0x7fbf2413f710 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:31.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 10 monclient(hunting): picked mon.noname-a con 0x7fbf241430c0 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:31.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf241390f0 con 0x7fbf241430c0 2024-03-25T18:50:31.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf24138dd0 con 0x7fbf240a98f0 2024-03-25T18:50:31.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf241346b0 con 0x7fbf2413f710 2024-03-25T18:50:31.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:31.482 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.477+0000 7fbf28a4a640 10 monclient(hunting): authenticate will time out at 6737.666091s 2024-03-25T18:50:31.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.487+0000 7fbf2259c640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7fbf240a98f0 0x7fbf24137e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:43778/0 (socket says 172.21.15.32:43778) 2024-03-25T18:50:31.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.487+0000 7fbf2259c640 1 -- 172.21.15.32:0/2657880175 learned_addr learned my addr 172.21.15.32:0/2657880175 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:31.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.487+0000 7fbf12ffd640 10 client.?.objecter ms_handle_connect 0x7fbf240a98f0 2024-03-25T18:50:31.487 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.487+0000 7fbf12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:31.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.496+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3902976881 0 0) 0x7fbf24138dd0 con 0x7fbf240a98f0 2024-03-25T18:50:31.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.496+0000 7fbf12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:50:31.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.496+0000 7fbf12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:31.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.496+0000 7fbf12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:31.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.496+0000 7fbf12ffd640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:50:31.496 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.496+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf04004eb0 con 0x7fbf240a98f0 2024-03-25T18:50:31.497 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.497+0000 7fbf12ffd640 10 client.?.objecter ms_handle_connect 0x7fbf2413f710 2024-03-25T18:50:31.497 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.497+0000 7fbf12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:31.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.501+0000 7fbf12ffd640 10 client.?.objecter ms_handle_connect 0x7fbf241430c0 2024-03-25T18:50:31.501 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.501+0000 7fbf12ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:31.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.509+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1328410847 0 0) 0x7fbf241346b0 con 0x7fbf2413f710 2024-03-25T18:50:31.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.509+0000 7fbf12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:50:31.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.509+0000 7fbf12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:31.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.509+0000 7fbf12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:31.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.509+0000 7fbf12ffd640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:50:31.509 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.509+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf24138dd0 con 0x7fbf2413f710 2024-03-25T18:50:31.510 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.510+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2483157262 0 0) 0x7fbf04004eb0 con 0x7fbf240a98f0 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.510+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf241346b0 con 0x7fbf240a98f0 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.510+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbf14001670 con 0x7fbf240a98f0 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.510+0000 7fbf12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.510+0000 7fbf12ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.510+0000 7fbf12ffd640 10 monclient(hunting): dump: 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:31.511 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.510+0000 7fbf12ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.511+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685759939 0 0) 0x7fbf241390f0 con 0x7fbf241430c0 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.511+0000 7fbf12ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.511+0000 7fbf12ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.511+0000 7fbf12ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.511+0000 7fbf12ffd640 10 monclient(hunting): my global_id is 4749 2024-03-25T18:50:31.512 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.511+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf04004eb0 con 0x7fbf241430c0 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.520+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.2 v1:172.21.15.32:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 475721108 0 0) 0x7fbf24138dd0 con 0x7fbf2413f710 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.520+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.32:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf241390f0 con 0x7fbf2413f710 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.520+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.2 v1:172.21.15.32:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbf18001950 con 0x7fbf2413f710 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.520+0000 7fbf12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.520+0000 7fbf12ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.520+0000 7fbf12ffd640 10 monclient(hunting): dump: 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.520 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.521 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:31.521 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:31.521 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:31.521 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:31.521 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:31.521 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.523+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.0 v1:172.21.15.32:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1598672948 0 0) 0x7fbf04004eb0 con 0x7fbf241430c0 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.523+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.32:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf24138dd0 con 0x7fbf241430c0 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.523+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.0 v1:172.21.15.32:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbf1c001bd0 con 0x7fbf241430c0 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.523+0000 7fbf12ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.523+0000 7fbf12ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.523+0000 7fbf12ffd640 10 monclient(hunting): dump: 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:31.524 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:31.525 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:31.525 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:31.526 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1324086005 0 0) 0x7fbf241346b0 con 0x7fbf240a98f0 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 >> v1:172.21.15.32:6790/0 conn(0x7fbf2413f710 legacy=0x7fbf241386c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 >> v1:172.21.15.32:6789/0 conn(0x7fbf241430c0 legacy=0x7fbf24133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 1 monclient: found mon.b 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf24134840 con 0x7fbf240a98f0 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 10 monclient: _finish_auth 0 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf12ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 5 monclient: authenticate success, global_id 4723 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 10 monclient: _renew_subs 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:31.527 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf24134a90 con 0x7fbf240a98f0 2024-03-25T18:50:31.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 10 client.4723.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:50:31.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 10 monclient: _renew_subs 2024-03-25T18:50:31.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:31.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf241356f0 con 0x7fbf240a98f0 2024-03-25T18:50:31.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.526+0000 7fbf28a4a640 1 librados: init done 2024-03-25T18:50:31.528 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.527+0000 7fbf28a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:31.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.527+0000 7fbf28a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:31.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.527+0000 7fbf28a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:31.529 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.527+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf24139280 con 0x7fbf240a98f0 2024-03-25T18:50:31.804 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.804+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf140038a0 con 0x7fbf240a98f0 2024-03-25T18:50:31.804 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.804+0000 7fbf12ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:31.804 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.804+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7fbf14003ad0 con 0x7fbf240a98f0 2024-03-25T18:50:31.804 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.804+0000 7fbf12ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:31.804 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.804+0000 7fbf12ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.804+0000 7fbf12ffd640 10 monclient: dump: 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:31.805 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.804+0000 7fbf20d99640 4 set_mon_vals no callback set 2024-03-25T18:50:31.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.805+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7fbf14003cf0 con 0x7fbf240a98f0 2024-03-25T18:50:31.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.805+0000 7fbf21d9b640 10 monclient: get_auth_request con 0x7fbf0406e160 auth_method 0 2024-03-25T18:50:31.806 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.806+0000 7fbf12ffd640 10 client.4723.objecter ms_handle_connect 0x7fbf0406e160 2024-03-25T18:50:31.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.811+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7fbf14054770 con 0x7fbf240a98f0 2024-03-25T18:50:31.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.811+0000 7fbf12ffd640 10 client.4723.objecter ms_dispatch 0x7fbf24000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:50:31.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.811+0000 7fbf12ffd640 3 client.4723.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:50:31.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.811+0000 7fbf12ffd640 3 client.4723.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:50:31.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.811+0000 7fbf12ffd640 20 client.4723.objecter dump_active .. 0 homeless 2024-03-25T18:50:31.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.811+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf14050280 con 0x7fbf240a98f0 2024-03-25T18:50:31.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.811+0000 7fbf12ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:31.811 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:31.811+0000 7fbf12ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:50:32.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.074+0000 7fbf28a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:32.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.074+0000 7fbf28a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:32.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.074+0000 7fbf28a4a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:32.075 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.074+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf240627b0 con 0x7fbf240a98f0 2024-03-25T18:50:32.097 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.097+0000 7fbf12ffd640 1 -- 172.21.15.32:0/2657880175 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7fbf140521b0 con 0x7fbf240a98f0 2024-03-25T18:50:32.097 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.097+0000 7fbf12ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:32.098 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.097+0000 7fbf12ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:50:32.098 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:50:32.098 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:50:32.100 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 10 librados: watch_flush enter 2024-03-25T18:50:32.100 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 10 librados: watch_flush exit 2024-03-25T18:50:32.100 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 20 client.4723.objecter shutdown clearing up homeless session... 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 10 client.4723.objecter successfully canceled tick 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 >> v1:172.21.15.136:6800/1141502808 conn(0x7fbf0406e160 legacy=0x7fbf04070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 10 monclient: shutdown 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 >> v1:172.21.15.136:6789/0 conn(0x7fbf240a98f0 legacy=0x7fbf24137e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 shutdown_connections 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 >> 172.21.15.32:0/2657880175 conn(0x7fbf240b7ea0 msgr2=0x7fbf24074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 shutdown_connections 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 1 -- 172.21.15.32:0/2657880175 wait complete. 2024-03-25T18:50:32.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:32.100+0000 7fbf28a4a640 1 librados: shutdown 2024-03-25T18:50:32.112 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:50:38.920 DEBUG:teuthology.orchestra.run.smithi032:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:50:39.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.035+0000 7f6a34aba640 10 monclient: get_monmap_and_config 2024-03-25T18:50:39.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.036+0000 7f6a34aba640 10 monclient: build_initial_monmap 2024-03-25T18:50:39.036 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.036+0000 7f6a34aba640 10 monclient: monmap: 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:50:39.036488+0000 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:50:39.036488+0000 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-c 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-b 2024-03-25T18:50:39.037 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:39.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.036+0000 7f6a34aba640 1 Processor -- start 2024-03-25T18:50:39.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 1 -- start start 2024-03-25T18:50:39.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 10 monclient: init 2024-03-25T18:50:39.039 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 10 monclient(hunting): picked mon.noname-b con 0x7f6a300b4d60 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 10 monclient(hunting): picked mon.noname-a con 0x7f6a300b1730 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 10 monclient(hunting): picked mon.noname-c con 0x7f6a3014be90 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300af940 con 0x7f6a300b1730 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300b0a70 con 0x7f6a3014be90 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a300b0d50 con 0x7f6a300b4d60 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:39.040 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.037+0000 7f6a34aba640 10 monclient(hunting): authenticate will time out at 6745.226282s 2024-03-25T18:50:39.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.044+0000 7f6a2e59c640 1 --1- >> v1:172.21.15.32:6790/0 conn(0x7f6a300b4d60 0x7f6a3013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.32:6790/0 says I am v1:172.21.15.32:53038/0 (socket says 172.21.15.32:53038) 2024-03-25T18:50:39.044 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.044+0000 7f6a2e59c640 1 -- 172.21.15.32:0/1311092254 learned_addr learned my addr 172.21.15.32:0/1311092254 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:39.051 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.051+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1032415854 0 0) 0x7f6a300b0a70 con 0x7f6a3014be90 2024-03-25T18:50:39.051 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.051+0000 7f6a2dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:50:39.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.051+0000 7f6a2dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:39.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.051+0000 7f6a2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:39.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.051+0000 7f6a2dd9b640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:50:39.052 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.051+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a10003520 con 0x7f6a3014be90 2024-03-25T18:50:39.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.058+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 108795647 0 0) 0x7f6a300b0d50 con 0x7f6a300b4d60 2024-03-25T18:50:39.058 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.058+0000 7f6a2dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:50:39.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.058+0000 7f6a2dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:39.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.058+0000 7f6a2dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:39.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.058+0000 7f6a2dd9b640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:50:39.059 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.058+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a300b0a70 con 0x7f6a300b4d60 2024-03-25T18:50:39.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 804961391 0 0) 0x7f6a10003520 con 0x7f6a3014be90 2024-03-25T18:50:39.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 >> v1:172.21.15.32:6790/0 conn(0x7f6a300b4d60 legacy=0x7f6a3013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:39.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 >> v1:172.21.15.32:6789/0 conn(0x7f6a300b1730 legacy=0x7f6a300b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:39.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:50:39.071 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:50:39.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:39.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.136:6789/0 2024-03-25T18:50:39.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a300b1030 con 0x7f6a3014be90 2024-03-25T18:50:39.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:50:39.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:50:39.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:39.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6a28001a10 con 0x7f6a3014be90 2024-03-25T18:50:39.072 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a34aba640 5 monclient: authenticate success, global_id 4726 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a34aba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a2dd9b640 10 monclient: dump: 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:39.073 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:39.074 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:39.074 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:39.074 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:39.074 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.070+0000 7f6a34aba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:50:39.079 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.079+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 <== mon.1 v1:172.21.15.136:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a28001b70 con 0x7f6a3014be90 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.079+0000 7f6a2dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.079+0000 7f6a2dd9b640 1 -- 172.21.15.32:0/1311092254 <== mon.1 v1:172.21.15.136:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6a28001f80 con 0x7f6a3014be90 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.079+0000 7f6a2dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.079+0000 7f6a2dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.079+0000 7f6a2dd9b640 10 monclient: dump: 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:39.080 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:39.081 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:39.081 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:39.081 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:39.081 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:39.081 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:39.081 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:39.081 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.079+0000 7f6a34aba640 10 monclient: get_monmap_and_config success 2024-03-25T18:50:39.081 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.079+0000 7f6a34aba640 4 set_mon_vals no callback set 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.080+0000 7f6a34aba640 10 monclient: shutdown 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.080+0000 7f6a34aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.080+0000 7f6a34aba640 1 -- 172.21.15.32:0/1311092254 >> v1:172.21.15.136:6789/0 conn(0x7f6a3014be90 legacy=0x7f6a3014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.080+0000 7f6a34aba640 1 -- 172.21.15.32:0/1311092254 shutdown_connections 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.080+0000 7f6a34aba640 1 -- 172.21.15.32:0/1311092254 >> 172.21.15.32:0/1311092254 conn(0x7f6a300617a0 msgr2=0x7f6a30061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.080+0000 7f6a34aba640 1 -- 172.21.15.32:0/1311092254 shutdown_connections 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.080+0000 7f6a34aba640 1 -- 172.21.15.32:0/1311092254 wait complete. 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 10 monclient: build_initial_monmap 2024-03-25T18:50:39.084 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 10 monclient: monmap: 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:epoch 0 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:last_changed 0.000000 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:created 0.000000 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 0 (unknown) 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 1 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.noname-a 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.noname-b 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.noname-c 2024-03-25T18:50:39.085 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 1 librados: starting msgr at 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 1 librados: starting objecter 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 1 Processor -- start 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 1 -- start start 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 1 librados: setting wanted keys 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 1 librados: calling monclient init 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.081+0000 7f6a34aba640 10 monclient: init 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 10 monclient: _reopen_session rank -1 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:50:39.086 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 10 monclient(hunting): picked mon.noname-c con 0x7f6a300b1730 addr v1:172.21.15.32:6790/0 2024-03-25T18:50:39.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 10 monclient(hunting): picked mon.noname-b con 0x7f6a300b4d60 addr v1:172.21.15.136:6789/0 2024-03-25T18:50:39.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 10 monclient(hunting): picked mon.noname-a con 0x7f6a3014be90 addr v1:172.21.15.32:6789/0 2024-03-25T18:50:39.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 1 -- --> v1:172.21.15.32:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a30140380 con 0x7f6a3014be90 2024-03-25T18:50:39.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 1 -- --> v1:172.21.15.136:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a30140660 con 0x7f6a300b4d60 2024-03-25T18:50:39.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 1 -- --> v1:172.21.15.32:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a30141e30 con 0x7f6a300b1730 2024-03-25T18:50:39.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 10 monclient(hunting): _renew_subs 2024-03-25T18:50:39.087 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.082+0000 7f6a34aba640 10 monclient(hunting): authenticate will time out at 6745.270998s 2024-03-25T18:50:39.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.093+0000 7f6a27fff640 1 --1- >> v1:172.21.15.136:6789/0 conn(0x7f6a300b4d60 0x7f6a30142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.136:6789/0 says I am v1:172.21.15.32:35896/0 (socket says 172.21.15.32:35896) 2024-03-25T18:50:39.093 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.093+0000 7f6a27fff640 1 -- 172.21.15.32:0/3172800599 learned_addr learned my addr 172.21.15.32:0/3172800599 (peer_addr_for_me v1:172.21.15.32:0/0) 2024-03-25T18:50:39.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.096+0000 7f6a26ffd640 10 client.?.objecter ms_handle_connect 0x7f6a300b4d60 2024-03-25T18:50:39.096 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.096+0000 7f6a26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:39.100 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.100+0000 7f6a26ffd640 10 client.?.objecter ms_handle_connect 0x7f6a300b1730 2024-03-25T18:50:39.100 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.100+0000 7f6a26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:39.100 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.100+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313746388 0 0) 0x7f6a30140660 con 0x7f6a300b4d60 2024-03-25T18:50:39.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.100+0000 7f6a26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:50:39.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.100+0000 7f6a26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:39.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.100+0000 7f6a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:39.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.100+0000 7f6a26ffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:50:39.101 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.100+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.136:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a14004eb0 con 0x7f6a300b4d60 2024-03-25T18:50:39.103 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.103+0000 7f6a26ffd640 10 client.?.objecter ms_handle_connect 0x7f6a3014be90 2024-03-25T18:50:39.103 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.103+0000 7f6a26ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:50:39.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.112+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3285409960 0 0) 0x7f6a14004eb0 con 0x7f6a300b4d60 2024-03-25T18:50:39.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.112+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.136:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a30140660 con 0x7f6a300b4d60 2024-03-25T18:50:39.112 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.112+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6a20001a30 con 0x7f6a300b4d60 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.112+0000 7f6a26ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.112+0000 7f6a26ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.112+0000 7f6a26ffd640 10 monclient(hunting): dump: 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:39.113 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:39.114 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:39.114 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:39.114 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:39.114 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.112+0000 7f6a26ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:50:39.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.118+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.0 v1:172.21.15.32:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883865785 0 0) 0x7f6a30140380 con 0x7f6a3014be90 2024-03-25T18:50:39.118 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.118+0000 7f6a26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:50:39.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.118+0000 7f6a26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:39.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.118+0000 7f6a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:39.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.118+0000 7f6a26ffd640 10 monclient(hunting): my global_id is 4755 2024-03-25T18:50:39.119 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.118+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.32:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a14004eb0 con 0x7f6a3014be90 2024-03-25T18:50:39.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.120+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.2 v1:172.21.15.32:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348918486 0 0) 0x7f6a30141e30 con 0x7f6a300b1730 2024-03-25T18:50:39.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.120+0000 7f6a26ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:50:39.120 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.120+0000 7f6a26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:50:39.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.120+0000 7f6a26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:50:39.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.120+0000 7f6a26ffd640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:50:39.121 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.120+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.32:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a30140380 con 0x7f6a300b1730 2024-03-25T18:50:39.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 157967248 0 0) 0x7f6a30140660 con 0x7f6a300b4d60 2024-03-25T18:50:39.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 >> v1:172.21.15.32:6790/0 conn(0x7f6a300b1730 legacy=0x7f6a30138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:39.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 >> v1:172.21.15.32:6789/0 conn(0x7f6a3014be90 legacy=0x7f6a301e2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:39.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:50:39.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 1 monclient: found mon.b 2024-03-25T18:50:39.124 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.136:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a30142110 con 0x7f6a300b4d60 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 10 monclient: _finish_auth 0 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 5 monclient: authenticate success, global_id 4729 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 10 monclient: _renew_subs 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.136:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a30140e60 con 0x7f6a300b4d60 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 10 monclient: _renew_subs 2024-03-25T18:50:39.125 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:39.126 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.136:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a30142c40 con 0x7f6a300b4d60 2024-03-25T18:50:39.126 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.123+0000 7f6a34aba640 1 librados: init done 2024-03-25T18:50:39.127 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.125+0000 7f6a34aba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:39.127 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.125+0000 7f6a34aba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:39.127 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.125+0000 7f6a34aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:39.127 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.125+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a300627b0 con 0x7f6a300b4d60 2024-03-25T18:50:39.390 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.390+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a20001cb0 con 0x7f6a300b4d60 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.390+0000 7f6a26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.390+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2945387820 0 0) 0x7f6a20003a60 con 0x7f6a300b4d60 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.390+0000 7f6a26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.390+0000 7f6a26ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.390+0000 7f6a26ffd640 10 monclient: dump: 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:epoch 1 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:fsid d7917469-fe02-4321-912b-a3ddb3c8ed96 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:last_changed 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:created 2024-03-25T18:38:16.671661+0000 2024-03-25T18:50:39.391 INFO:teuthology.misc.health.smithi032.stderr:min_mon_release 19 (squid) 2024-03-25T18:50:39.392 INFO:teuthology.misc.health.smithi032.stderr:election_strategy: 3 2024-03-25T18:50:39.392 INFO:teuthology.misc.health.smithi032.stderr:0: v1:172.21.15.32:6789/0 mon.a 2024-03-25T18:50:39.392 INFO:teuthology.misc.health.smithi032.stderr:1: v1:172.21.15.136:6789/0 mon.b 2024-03-25T18:50:39.392 INFO:teuthology.misc.health.smithi032.stderr:2: v1:172.21.15.32:6790/0 mon.c 2024-03-25T18:50:39.392 INFO:teuthology.misc.health.smithi032.stderr: 2024-03-25T18:50:39.392 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.390+0000 7f6a2d59a640 4 set_mon_vals no callback set 2024-03-25T18:50:39.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.391+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 309164751 0 0) 0x7f6a20003c40 con 0x7f6a300b4d60 2024-03-25T18:50:39.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.392+0000 7f6a2e59c640 10 monclient: get_auth_request con 0x7f6a1406de60 auth_method 0 2024-03-25T18:50:39.393 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.392+0000 7f6a26ffd640 10 client.4729.objecter ms_handle_connect 0x7f6a1406de60 2024-03-25T18:50:39.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.396+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2451301444 0 0) 0x7f6a20054870 con 0x7f6a300b4d60 2024-03-25T18:50:39.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.396+0000 7f6a26ffd640 10 client.4729.objecter ms_dispatch 0x7f6a30000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:50:39.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.396+0000 7f6a26ffd640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:50:39.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.396+0000 7f6a26ffd640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:50:39.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.396+0000 7f6a26ffd640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T18:50:39.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.396+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a20050380 con 0x7f6a300b4d60 2024-03-25T18:50:39.397 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.396+0000 7f6a26ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:50:39.398 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.396+0000 7f6a26ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:50:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.624+0000 7f6a34aba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.624+0000 7f6a34aba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.624+0000 7f6a34aba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.136:6789/0 2024-03-25T18:50:39.624 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.624+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 --> v1:172.21.15.136:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a30140810 con 0x7f6a300b4d60 2024-03-25T18:50:39.666 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:51.994 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:39.675 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:52.003 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:39.717 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:11:52.045 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:39.724 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:52.052 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:39.735 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:52.063 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:39.735 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:52.052 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:39.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.757+0000 7f6a26ffd640 1 -- 172.21.15.32:0/3172800599 <== mon.1 v1:172.21.15.136:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 1970407246) 0x7f6a30140810 con 0x7f6a300b4d60 2024-03-25T18:50:39.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.757+0000 7f6a26ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:50:39.758 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.757+0000 7f6a26ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:50:39.758 INFO:teuthology.misc.health.smithi032.stdout: 2024-03-25T18:50:39.759 INFO:teuthology.misc.health.smithi032.stdout:{"epoch":11,"fsid":"d7917469-fe02-4321-912b-a3ddb3c8ed96","created":"2024-03-25T18:39:10.234522+0000","modified":"2024-03-25T18:49:14.062522+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:49:14.062522+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"3c5e9771-b868-42d0-bad2-4d4e060105f4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"264b5aa3-cfe9-4b37-b248-601a91ea8907","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"7d429f95-7ad6-426f-bc51-b9e2a750a918","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"9fa2130c-1583-40e1-97c5-dc72314a5358","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d4d8f0df-36be-4807-b303-9edc2bb3b34e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"c05dfe41-a0ff-4367-a4e9-0173990d2eaf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ab217fa6-09ba-4227-87a1-9e5d660603d1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"6320cd08-17e0-4bc6-8b8e-9d8341a59cd1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:50:39.760 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.759+0000 7f6a34aba640 10 librados: watch_flush enter 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.759+0000 7f6a34aba640 10 librados: watch_flush exit 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.759+0000 7f6a34aba640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.759+0000 7f6a34aba640 10 client.4729.objecter successfully canceled tick 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 >> v1:172.21.15.136:6800/1141502808 conn(0x7f6a1406de60 legacy=0x7f6a14070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 10 monclient: shutdown 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 >> v1:172.21.15.136:6789/0 conn(0x7f6a300b4d60 legacy=0x7f6a30142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 shutdown_connections 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 >> 172.21.15.32:0/3172800599 conn(0x7f6a300617a0 msgr2=0x7f6a300b84e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:50:39.761 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 shutdown_connections 2024-03-25T18:50:39.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 1 -- 172.21.15.32:0/3172800599 wait complete. 2024-03-25T18:50:39.762 INFO:teuthology.misc.health.smithi032.stderr:2024-03-25T18:50:39.760+0000 7f6a34aba640 1 librados: shutdown 2024-03-25T18:50:39.764 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:52.081 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:39.773 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:50:39.773 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:50:39.776 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:50:39.776 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:50:39.776 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:50:39.783 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:11:52.111 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:39.801 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:11:52.118 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:41.705 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:11:25.965 35184== 2024-03-25T18:50:41.705 INFO:tasks.ceph.osd.0.smithi032.stderr:==00:00:11:25.965 35184== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:50:41.746 INFO:tasks.ceph.osd.0.smithi032.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:50:45.879 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:50:45.879 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:50:45.880 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:50:45.880 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:50:45.880 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:50:47.903 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:11:32.175 35183== 2024-03-25T18:50:47.903 INFO:tasks.ceph.osd.1.smithi032.stderr:==00:00:11:32.175 35183== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:50:47.940 INFO:tasks.ceph.osd.1.smithi032.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:50:51.981 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:50:51.982 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:50:51.982 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:50:51.982 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:50:51.982 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:50:53.810 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:11:38.066 35185== 2024-03-25T18:50:53.810 INFO:tasks.ceph.osd.2.smithi032.stderr:==00:00:11:38.066 35185== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:50:53.847 INFO:tasks.ceph.osd.2.smithi032.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:50:58.084 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:50:58.084 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:50:58.084 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:50:58.085 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:50:58.085 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:50:59.507 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:12:11.834 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:59.519 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:12:11.847 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:59.542 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:12:11.870 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:59.554 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:12:11.871 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:59.556 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:12:11.873 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:59.563 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:12:11.891 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:59.571 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:12:11.899 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:59.584 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:12:11.901 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:50:59.594 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:12:11.922 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:00.067 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:11:44.314 35186== 2024-03-25T18:51:00.067 INFO:tasks.ceph.osd.3.smithi032.stderr:==00:00:11:44.314 35186== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:51:00.107 INFO:tasks.ceph.osd.3.smithi032.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:51:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:51:04.187 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:51:04.187 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:51:04.187 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:51:04.188 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:51:06.273 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:11:50.548 35405== 2024-03-25T18:51:06.273 INFO:tasks.ceph.osd.4.smithi136.stderr:==00:00:11:50.548 35405== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:51:06.308 INFO:tasks.ceph.osd.4.smithi136.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:51:10.289 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:51:10.289 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi136 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:51:10.290 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:51:10.290 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:51:10.290 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:51:12.273 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:11:56.545 35406== 2024-03-25T18:51:12.274 INFO:tasks.ceph.osd.5.smithi136.stderr:==00:00:11:56.545 35406== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:51:12.306 INFO:tasks.ceph.osd.5.smithi136.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:51:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:51:16.393 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi136 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:51:16.393 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:51:16.394 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:51:16.394 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:51:18.087 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:12:02.358 35407== 2024-03-25T18:51:18.087 INFO:tasks.ceph.osd.6.smithi136.stderr:==00:00:12:02.358 35407== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:51:18.123 INFO:tasks.ceph.osd.6.smithi136.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:51:19.508 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:12:31.824 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:19.509 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:12:31.837 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:19.511 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:12:31.839 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:19.512 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:12:31.829 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:19.529 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:12:31.857 34867== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:19.529 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:12:31.846 34859== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:19.559 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:12:31.887 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:19.564 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:12:31.893 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:19.580 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:12:31.908 34871== Warning: unimplemented fcntl command: 1036 2024-03-25T18:51:22.495 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:51:22.496 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi136 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:51:22.496 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:51:22.496 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:51:22.496 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:51:24.065 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:12:08.331 35408== 2024-03-25T18:51:24.065 INFO:tasks.ceph.osd.7.smithi136.stderr:==00:00:12:08.331 35408== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:51:24.095 INFO:tasks.ceph.osd.7.smithi136.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:51:28.598 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:51:28.599 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi136 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:51:28.599 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:51:28.600 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:51:28.600 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:51:28.600 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:51:28.624 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:51:28.624 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:51:28.624 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:51:28.625 INFO:tasks.ceph.mgr.x.smithi136.stderr:2024-03-25T18:51:28.625+0000 7f97b4d14640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34857) UID: 0 2024-03-25T18:51:28.651 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:51:28.651 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:51:28.651 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:51:28.651 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:51:28.680 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-25T18:51:28.668+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 34836) UID: 0 2024-03-25T18:51:28.680 INFO:tasks.ceph.mon.a.smithi032.stderr:2024-03-25T18:51:28.670+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:51:29.102 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:12:41.429 34867== 2024-03-25T18:51:29.102 INFO:tasks.ceph.mon.a.smithi032.stderr:==00:00:12:41.429 34867== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:51:29.111 INFO:tasks.ceph.mon.a.smithi032.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:51:34.753 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:51:34.754 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:51:34.754 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:51:34.755 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:51:34.755 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:51:34.771 INFO:tasks.ceph.mon.c.smithi032.stderr:2024-03-25T18:51:34.758+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 34838) UID: 0 2024-03-25T18:51:34.772 INFO:tasks.ceph.mon.c.smithi032.stderr:2024-03-25T18:51:34.760+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:53:58.242 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:15:10.569 34871== 2024-03-25T18:53:58.242 INFO:tasks.ceph.mon.c.smithi032.stderr:==00:00:15:10.569 34871== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:53:58.250 INFO:tasks.ceph.mon.c.smithi032.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:54:01.189 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:54:01.189 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:54:01.190 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:54:01.190 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:54:01.190 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:54:01.206 INFO:tasks.ceph.mon.b.smithi136.stderr:2024-03-25T18:54:01.195+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34855) UID: 0 2024-03-25T18:54:01.207 INFO:tasks.ceph.mon.b.smithi136.stderr:2024-03-25T18:54:01.196+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:54:01.863 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:15:14.180 34859== 2024-03-25T18:54:01.864 INFO:tasks.ceph.mon.b.smithi136.stderr:==00:00:15:14.180 34859== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:54:01.874 INFO:tasks.ceph.mon.b.smithi136.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:54:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:54:07.293 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi136 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:54:07.293 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:54:07.294 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:54:07.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:54:07.344 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:54:07.344 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:54:07.456 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:54:07.456 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:54:07.530 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:54:07.530 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:54:07.589 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:54:07.589 DEBUG:teuthology.orchestra.run.smithi032:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:54:07.656 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:54:07.656 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:54:07.756 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:54:07.756 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:54:07.812 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:54:07.812 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:54:07.869 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:54:07.869 DEBUG:teuthology.orchestra.run.smithi136:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:54:07.927 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:54:07.929 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/data/mon.a.tgz 2024-03-25T18:54:07.929 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-25T18:54:07.944 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.H12TCrJ9kg 2024-03-25T18:54:07.944 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.H12TCrJ9kg 2024-03-25T18:54:08.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.H12TCrJ9kg 2024-03-25T18:54:08.132 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.H12TCrJ9kg is 133KB 2024-03-25T18:54:08.184 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.H12TCrJ9kg 2024-03-25T18:54:08.198 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/data/mon.c.tgz 2024-03-25T18:54:08.198 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-03-25T18:54:08.252 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.wEDDKkjBqO 2024-03-25T18:54:08.252 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.wEDDKkjBqO 2024-03-25T18:54:08.339 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.wEDDKkjBqO 2024-03-25T18:54:08.423 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.wEDDKkjBqO is 133KB 2024-03-25T18:54:08.474 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.wEDDKkjBqO 2024-03-25T18:54:08.489 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/data/mon.b.tgz 2024-03-25T18:54:08.489 DEBUG:teuthology.orchestra.run.smithi136:> mktemp 2024-03-25T18:54:08.506 INFO:teuthology.orchestra.run.smithi136.stdout:/tmp/tmp.WgnIU2wGdD 2024-03-25T18:54:08.507 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.WgnIU2wGdD 2024-03-25T18:54:08.609 DEBUG:teuthology.orchestra.run.smithi136:> sudo chmod 0666 /tmp/tmp.WgnIU2wGdD 2024-03-25T18:54:08.696 DEBUG:teuthology.orchestra.remote:smithi136:/tmp/tmp.WgnIU2wGdD is 134KB 2024-03-25T18:54:08.747 DEBUG:teuthology.orchestra.run.smithi136:> rm -fr /tmp/tmp.WgnIU2wGdD 2024-03-25T18:54:08.762 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:54:08.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:54:08.764 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:54:09.150 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:54:09.391 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:54:09.393 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/remote/smithi032/crash 2024-03-25T18:54:09.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:54:09.421 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/remote/smithi136/crash 2024-03-25T18:54:09.422 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:54:09.448 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:54:09.448 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:54:09.465 DEBUG:teuthology.orchestra.run.smithi136:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:54:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:54:09.537 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/valgrind/osd.6.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35135.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35306.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35135.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35265.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35095.log 2024-03-25T18:54:09.538 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35135.log.gz 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35265.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.35306.log.gz 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35095.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35265.log.gz 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35055.log 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35095.log.gz 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35055.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35015.log 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35055.log.gz 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35175.log 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35015.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35175.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35015.log.gz 2024-03-25T18:54:09.539 INFO:teuthology.orchestra.run.smithi136.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35215.log 2024-03-25T18:54:09.540 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35175.log.gz 2024-03-25T18:54:09.540 INFO:teuthology.orchestra.run.smithi136.stderr: --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:54:09.540 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-client.admin.35215.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:54:09.540 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35215.log.gz 2024-03-25T18:54:09.540 INFO:teuthology.orchestra.run.smithi136.stderr:/var/log/ceph/ceph-mon.b.log: 87.4% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:54:09.541 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41452.log 2024-03-25T18:54:09.541 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:54:09.541 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41452.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41803.log 2024-03-25T18:54:09.541 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36165.log 2024-03-25T18:54:09.541 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41452.log.gz 2024-03-25T18:54:09.541 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41686.log 2024-03-25T18:54:09.541 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.36165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39345.log 2024-03-25T18:54:09.541 INFO:teuthology.orchestra.run.smithi032.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41803.log.gz 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41686.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39267.log 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36165.log.gz 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39345.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41686.log.gzgzip 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.34878.log 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39267.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39345.log.gz 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39462.log 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.34878.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41062.log 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39267.log.gz 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39462.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38877.log 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.34878.log.gz 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41062.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39462.log.gz 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35182.log 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38877.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40086.log 2024-03-25T18:54:09.542 INFO:teuthology.orchestra.run.smithi032.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41062.log.gz 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35182.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38877.log.gz 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40086.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.35182.log.gz 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mon.c.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40086.log.gz 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/osd.0.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:54:09.543 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38479.log 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39969.log 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40008.log 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39969.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.38479.log.gz 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39306.log 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40008.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39969.log.gz 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42001.log 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39306.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40906.log 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40008.log.gz 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.42001.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39306.log.gz 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40945.log 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39072.log 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40906.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.42001.log.gz 2024-03-25T18:54:09.544 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40945.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40633.log 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40906.log.gz 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39072.log: gzip -5 --verbose -- 87.1% /var/log/ceph/ceph-client.admin.40282.log -- replaced with /var/log/ceph/ceph-client.admin.40945.log.gz 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40633.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40867.log 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39072.log.gz 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40282.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40633.log.gz 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41961.log 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40867.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40282.log.gz 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38756.log 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41961.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39501.log 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40867.log.gz 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38756.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.41961.log.gz 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41764.log 2024-03-25T18:54:09.545 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39501.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38756.log.gz 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37135.log 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41764.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35209.log 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39501.log.gz 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.37135.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41569.log 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41764.log.gz 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35209.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.37135.log.gz 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38994.log 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41569.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41647.log 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.35209.log.gz 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38994.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41569.log.gz 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41647.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39540.log 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38994.log.gz 2024-03-25T18:54:09.546 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.audit.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41647.log.gz 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41296.log 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39540.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40789.log 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41296.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39540.log.gzgzip 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41491.log 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39735.log 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40789.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41296.log.gz 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40360.log 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40789.log.gz 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39735.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41491.log.gz 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40164.log 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39813.log 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39735.log.gz 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40164.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40360.log.gz 2024-03-25T18:54:09.547 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41140.log 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39813.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40164.log.gz 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40516.log 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41140.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41922.log 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39813.log.gz 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40516.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41140.log.gz 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29013.log 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41922.log: gzip -5 --verbose -- 86.9% /var/log/ceph/ceph-client.admin.40984.log 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.40516.log.gz 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.tmp-client.admin.29013.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39657.log 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29013.log.gz 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41922.log.gz 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40984.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39657.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41023.log 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40984.log.gz 2024-03-25T18:54:09.548 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.1.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39657.log.gz 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41883.log 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41023.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39384.log 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41883.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39228.log 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41023.log.gz 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39384.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41179.log 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41883.log.gz 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39228.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39384.log.gz 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40672.log 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41218.log 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39228.log.gz 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41179.log: /var/log/ceph/ceph-client.admin.40672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40399.log 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41218.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41179.log.gz 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40828.log 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40672.log.gz 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41218.log.gz 2024-03-25T18:54:09.549 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40399.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39930.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35324.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40828.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40399.log.gz 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40438.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40828.log.gz 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.35324.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41413.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39930.log.gz 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40438.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35324.log.gz 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40125.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40438.log.gz 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41413.log: /var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40477.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1%/var/log/ceph/ceph-client.admin.40125.log: -- replaced with /var/log/ceph/ceph-client.admin.41413.log.gz 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40594.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39423.log 2024-03-25T18:54:09.550 INFO:teuthology.orchestra.run.smithi032.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40125.log.gz 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40594.log: gzip -5 --verbose -- 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40477.log.gz 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.admin.40711.log 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39852.log 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40594.log.gz 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40711.log: gzip -5 --verbose -- 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39423.log.gz 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr: /var/log/ceph/ceph-client.admin.40047.log 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39852.log: gzip 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40711.log.gz 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38716.log 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40047.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39852.log.gz 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39189.log 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41608.log 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38716.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40047.log.gz 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40321.log 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38716.log.gz 2024-03-25T18:54:09.551 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41608.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39150.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39189.log.gz 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40321.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41608.log.gz 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39111.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39774.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39150.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40321.log.gz 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39111.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39774.log: 87.7% 86.0% -- replaced with /var/log/ceph/ceph-client.admin.39111.log.gz 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41842.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39150.log.gz 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41101.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mgr.y.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39774.log.gz 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41842.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40242.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41101.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41842.log.gz 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41257.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr:gzip/var/log/ceph/ceph-client.admin.40242.log: -5 86.9% --verbose -- /var/log/ceph/ceph-client.admin.40203.log 2024-03-25T18:54:09.552 INFO:teuthology.orchestra.run.smithi032.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41101.log.gz 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41257.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40555.log 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40242.log.gz 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40203.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41257.log.gz 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41335.log 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40555.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40203.log.gz 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41335.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40555.log.gz 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41530.log 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.0.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41335.log.gz 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39579.log 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41530.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40750.log 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39579.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41530.log.gz 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.40750.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39579.log.gz 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39696.log 2024-03-25T18:54:09.553 INFO:teuthology.orchestra.run.smithi032.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40750.log.gz 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41725.log 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39696.log.gz 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-osd.2.log: 92.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38838.log 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41725.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38916.log 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38838.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41725.log.gz 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38838.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39033.log 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38916.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38916.log.gz 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39891.log 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39033.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38797.log 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39891.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39033.log.gz 2024-03-25T18:54:09.554 INFO:teuthology.orchestra.run.smithi032.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39891.log.gz 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39618.log 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38797.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38797.log.gz 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr: 93.9% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41374.log 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.39618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38955.log 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.41374.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39618.log.gz 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-client.admin.38955.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41374.log.gz 2024-03-25T18:54:09.555 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/ceph-mon.c.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38955.log.gz 2024-03-25T18:54:09.600 INFO:teuthology.orchestra.run.smithi136.stderr: 92.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:54:10.323 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:54:10.395 INFO:teuthology.orchestra.run.smithi136.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:54:10.403 INFO:teuthology.orchestra.run.smithi136.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:54:10.423 INFO:teuthology.orchestra.run.smithi136.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:54:10.608 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:54:10.673 INFO:teuthology.orchestra.run.smithi136.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:54:10.732 INFO:teuthology.orchestra.run.smithi032.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:54:10.733 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:54:14.533 INFO:teuthology.orchestra.run.smithi032.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:54:14.664 INFO:teuthology.orchestra.run.smithi032.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:54:14.666 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-03-25T18:54:14.666 INFO:teuthology.orchestra.run.smithi032.stderr:real 0m5.189s 2024-03-25T18:54:14.666 INFO:teuthology.orchestra.run.smithi032.stderr:user 0m14.100s 2024-03-25T18:54:14.666 INFO:teuthology.orchestra.run.smithi032.stderr:sys 0m0.545s 2024-03-25T18:54:14.684 INFO:teuthology.orchestra.run.smithi136.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:54:14.686 INFO:teuthology.orchestra.run.smithi136.stderr: 2024-03-25T18:54:14.686 INFO:teuthology.orchestra.run.smithi136.stderr:real 0m5.179s 2024-03-25T18:54:14.686 INFO:teuthology.orchestra.run.smithi136.stderr:user 0m8.734s 2024-03-25T18:54:14.686 INFO:teuthology.orchestra.run.smithi136.stderr:sys 0m0.332s 2024-03-25T18:54:14.687 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:54:14.687 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/remote/smithi032/log 2024-03-25T18:54:14.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:54:17.081 DEBUG:teuthology.misc:Transferring archived files from smithi136:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/remote/smithi136/log 2024-03-25T18:54:17.082 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:54:18.578 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:54:18.790 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=446fd6f71ac2415cb359ad03ddaf5eb5 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:54:18.795 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:54:18.803 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:54:18.841 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:54:18.842 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:54:18.855 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:54:18.937 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:54:18.946 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:54:18.946 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:54:18.949 DEBUG:teuthology.orchestra.run.smithi136:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:54:18.963 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:54:18.965 INFO:teuthology.orchestra.run.smithi136.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:54:18.968 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:54:18.968 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-03-25T18:54:18.968 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +275us[ +276us] +/- 74ms 2024-03-25T18:54:18.968 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv02.front.sepia.ceph.com 3 6 377 4 -869us[ -870us] +/- 76ms 2024-03-25T18:54:18.969 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 2 +768us[ +768us] +/- 80ms 2024-03-25T18:54:18.969 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:54:18.970 INFO:teuthology.orchestra.run.smithi136.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:54:18.970 INFO:teuthology.orchestra.run.smithi136.stdout:=============================================================================== 2024-03-25T18:54:18.971 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 60 +241us[ +259us] +/- 74ms 2024-03-25T18:54:18.971 INFO:teuthology.orchestra.run.smithi136.stdout:^* hv02.front.sepia.ceph.com 3 6 377 1 -875us[ -858us] +/- 76ms 2024-03-25T18:54:18.971 INFO:teuthology.orchestra.run.smithi136.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 +769us[ +769us] +/- 80ms 2024-03-25T18:54:18.971 INFO:teuthology.orchestra.run.smithi136.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:54:18.971 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:54:18.979 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:54:18.980 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:54:19.007 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:54:19.084 DEBUG:teuthology.orchestra.run.smithi136:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:54:19.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:54:19.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:54:19.187 DEBUG:teuthology.orchestra.run.smithi136:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:54:19.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:54:19.219 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:54:19.228 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:54:19.248 INFO:teuthology.task.internal:Duration was 6597.166610 seconds 2024-03-25T18:54:19.248 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:54:19.273 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:54:19.274 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:54:19.276 DEBUG:teuthology.orchestra.run.smithi136:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:54:19.320 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:54:19.321 INFO:teuthology.orchestra.run.smithi136.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:54:19.573 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:54:19.574 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-03-25T18:54:19.574 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:54:19.594 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi136.front.sepia.ceph.com 2024-03-25T18:54:19.594 DEBUG:teuthology.orchestra.run.smithi136:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:54:19.620 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:54:19.620 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:54:19.637 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:54:19.688 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:54:19.689 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:54:19.707 DEBUG:teuthology.orchestra.run.smithi136:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:54:19.834 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:54:19.844 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:54:19.845 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:54:19.871 DEBUG:teuthology.orchestra.run.smithi136:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:54:19.899 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:54:19.908 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:54:19.914 DEBUG:teuthology.orchestra.run.smithi136:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:54:19.936 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-03-25T18:54:19.964 INFO:teuthology.orchestra.run.smithi136.stdout:kernel.core_pattern = core 2024-03-25T18:54:19.989 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:54:20.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:54:20.015 DEBUG:teuthology.orchestra.run.smithi136:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:54:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:54:20.043 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:54:20.052 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:54:20.053 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/remote/smithi032 2024-03-25T18:54:20.053 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:54:20.090 DEBUG:teuthology.misc:Transferring archived files from smithi136:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284/remote/smithi136 2024-03-25T18:54:20.090 DEBUG:teuthology.orchestra.run.smithi136:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:54:20.125 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:54:20.126 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:54:20.128 DEBUG:teuthology.orchestra.run.smithi136:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:54:20.175 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:54:20.185 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:54:20.185 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:54:20.242 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:54:20.242 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:54:20.245 DEBUG:teuthology.orchestra.run.smithi136:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:54:20.256 INFO:teuthology.orchestra.run.smithi032.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:54 /home/ubuntu/cephtest 2024-03-25T18:54:20.259 INFO:teuthology.orchestra.run.smithi136.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:54 /home/ubuntu/cephtest 2024-03-25T18:54:20.260 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:54:20.270 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:54:20.317 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:54:20.343 DEBUG:teuthology.nuke:shortname: smithi032 2024-03-25T18:54:20.343 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:54:20.371 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:55:50.025273', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLQ2cEhYIhasdCSbjjrbMgx4YxiqTKlERSjnvWz3CHmGkjcWVq30z9d3cBowRGiDQ5TgJ7u9R8Ifp+cYmJCQMH0='} 2024-03-25T18:54:20.375 DEBUG:teuthology.nuke:shortname: smithi136 2024-03-25T18:54:20.375 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:54:20.395 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi136.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621284', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:55:50.023666', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBIPAOGTiEUOU1pEwviVaRkC2dui5+QB0HeHVyRSXf0gqxq6Z2loYjA6tAp6VkzPXtjLroU92ffcKbBWZ8ZttYcw='} 2024-03-25T18:54:20.430 INFO:teuthology.orchestra.console.smithi032:Power off 2024-03-25T18:54:20.431 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:54:20.448 INFO:teuthology.orchestra.console.smithi136:Power off 2024-03-25T18:54:20.449 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:54:20.458 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-03-25T18:54:20.459 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:54:20.472 DEBUG:teuthology.orchestra.console.smithi136:power off output: Chassis Power Control: Down/Off 2024-03-25T18:54:20.472 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:54:20.481 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-03-25T18:54:20.496 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-25T18:54:24.482 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:54:24.497 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:54:24.592 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-03-25T18:54:24.608 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is on 2024-03-25T18:54:28.595 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:54:28.609 DEBUG:teuthology.orchestra.console.smithi136:pexpect command: ipmitool -H smithi136.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:54:28.705 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-03-25T18:54:28.720 DEBUG:teuthology.orchestra.console.smithi136:check power output: Chassis Power is off 2024-03-25T18:54:28.806 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-03-25T18:54:28.821 INFO:teuthology.orchestra.console.smithi136:Power off completed 2024-03-25T18:54:28.925 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 6597.166609764099 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=446fd6f71ac2415cb359ad03ddaf5eb5 status: fail success: false 2024-03-25T18:54:28.926 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:54:28.999 INFO:teuthology.run:FAIL